KVM/ARM changes for Linux v4.5 - Complete rewrite of the arm64 world switch in C, hopefully paving the way for more sharing with the 32bit code, better maintainability and easier integration of new features. Also smaller and slightly faster in some cases... - Support for 16bit VM identifiers - Various cleanups
Unified · Split
| @@ -49,24 +49,6 @@ specified through DTS. Following are the DTS used:- · 48 unmodified lines | ||
| 49 | 49 | The device tree documentation for the keystone machines are located at |
| 50 | 50 | Documentation/devicetree/bindings/arm/keystone/keystone.txt |
| 51 | 51 | |
| 52 | −Known issues & workaround | |
| 53 | −------------------------- | |
| 54 | − | |
| 55 | −Some of the device drivers used on keystone are re-used from that from | |
| 56 | −DaVinci and other TI SoCs. These device drivers may use clock APIs directly. | |
| 57 | −Some of the keystone specific drivers such as netcp uses run time power | |
| 58 | −management API instead to enable clock. As this API has limitations on | |
| 59 | −keystone, following workaround is needed to boot Linux. | |
| 60 | − | |
| 61 | − Add 'clk_ignore_unused' to the bootargs env variable in u-boot. Otherwise | |
| 62 | − clock frameworks will try to disable clocks that are unused and disable | |
| 63 | − the hardware. This is because netcp related power domain and clock | |
| 64 | − domains are enabled in u-boot as run time power management API currently | |
| 65 | − doesn't enable clocks for netcp due to a limitation. This workaround is | |
| 66 | − expected to be removed in the future when proper API support becomes | |
| 67 | − available. Until then, this work around is needed. | |
| 68 | − | |
| 69 | − | |
| 70 | 52 | Document Author |
| 71 | 53 | --------------- |
| 72 | 54 | Murali Karicheri <m-karicheri2@ti.com> |
| @@ -70,3 +70,6 @@ use_per_node_hctx=[0/1]: Default: 0 · 69 unmodified lines | ||
| 70 | 70 | parameter. |
| 71 | 71 | 1: The multi-queue block layer is instantiated with a hardware dispatch |
| 72 | 72 | queue for each CPU node in the system. |
| 73 | + | |
| 74 | +use_lightnvm=[0/1]: Default: 0 | |
| 75 | + Register device with LightNVM. Requires blk-mq to be used. | |
| @@ -11,6 +11,10 @@ Required properties: · 10 unmodified lines | ||
| 11 | 11 | 0 = active high |
| 12 | 12 | 1 = active low |
| 13 | 13 | |
| 14 | +Optional properties: | |
| 15 | +- little-endian : GPIO registers are used as little endian. If not | |
| 16 | + present registers are used as big endian by default. | |
| 17 | + | |
| 14 | 18 | Example: |
| 15 | 19 | |
| 16 | 20 | gpio0: gpio@1100 { |
| @@ -8,6 +8,11 @@ Required properties: · 7 unmodified lines | ||
| 8 | 8 | - phy-mode: See ethernet.txt file in the same directory |
| 9 | 9 | - clocks: a pointer to the reference clock for this device. |
| 10 | 10 | |
| 11 | +Optional properties: | |
| 12 | +- tx-csum-limit: maximum mtu supported by port that allow TX checksum. | |
| 13 | + Value is presented in bytes. If not used, by default 1600B is set for | |
| 14 | + "marvell,armada-370-neta" and 9800B for others. | |
| 15 | + | |
| 11 | 16 | Example: |
| 12 | 17 | |
| 13 | 18 | ethernet@d0070000 { |
| @@ -15,6 +20,7 @@ ethernet@d0070000 { · 1 unmodified line | ||
| 15 | 20 | reg = <0xd0070000 0x2500>; |
| 16 | 21 | interrupts = <8>; |
| 17 | 22 | clocks = <&gate_clk 4>; |
| 23 | + tx-csum-limit = <9800> | |
| 18 | 24 | status = "okay"; |
| 19 | 25 | phy = <&phy0>; |
| 20 | 26 | phy-mode = "rgmii-id"; |
| @@ -1,7 +1,9 @@ | ||
| 1 | 1 | * Temperature Sensor ADC (TSADC) on rockchip SoCs |
| 2 | 2 | |
| 3 | 3 | Required properties: |
| 4 | −- compatible : "rockchip,rk3288-tsadc" | |
| 4 | +- compatible : should be "rockchip,<name>-tsadc" | |
| 5 | + "rockchip,rk3288-tsadc": found on RK3288 SoCs | |
| 6 | + "rockchip,rk3368-tsadc": found on RK3368 SoCs | |
| 5 | 7 | - reg : physical base address of the controller and length of memory mapped |
| 6 | 8 | region. |
| 7 | 9 | - interrupts : The interrupt number to the cpu. The interrupt specifier format |
| @@ -318,7 +318,7 @@ M: Zhang Rui <rui.zhang@intel.com> · 317 unmodified lines | ||
| 318 | 318 | L: linux-acpi@vger.kernel.org |
| 319 | 319 | W: https://01.org/linux-acpi |
| 320 | 320 | S: Supported |
| 321 | −F: drivers/acpi/video.c | |
| 321 | +F: drivers/acpi/acpi_video.c | |
| 322 | 322 | |
| 323 | 323 | ACPI WMI DRIVER |
| 324 | 324 | L: platform-driver-x86@vger.kernel.org |
| @@ -1847,7 +1847,7 @@ S: Supported · 1522 unmodified lines | ||
| 1847 | 1847 | F: drivers/net/wireless/ath/ath6kl/ |
| 1848 | 1848 | |
| 1849 | 1849 | WILOCITY WIL6210 WIRELESS DRIVER |
| 1850 | −M: Vladimir Kondratiev <qca_vkondrat@qca.qualcomm.com> | |
| 1850 | +M: Maya Erez <qca_merez@qca.qualcomm.com> | |
| 1851 | 1851 | L: linux-wireless@vger.kernel.org |
| 1852 | 1852 | L: wil6210@qca.qualcomm.com |
| 1853 | 1853 | S: Supported |
| @@ -1931,7 +1931,7 @@ S: Supported · 77 unmodified lines | ||
| 1931 | 1931 | F: drivers/i2c/busses/i2c-at91.c |
| 1932 | 1932 | |
| 1933 | 1933 | ATMEL ISI DRIVER |
| 1934 | −M: Josh Wu <josh.wu@atmel.com> | |
| 1934 | +M: Ludovic Desroches <ludovic.desroches@atmel.com> | |
| 1935 | 1935 | L: linux-media@vger.kernel.org |
| 1936 | 1936 | S: Supported |
| 1937 | 1937 | F: drivers/media/platform/soc_camera/atmel-isi.c |
| @@ -1950,7 +1950,8 @@ S: Supported · 12 unmodified lines | ||
| 1950 | 1950 | F: drivers/net/ethernet/cadence/ |
| 1951 | 1951 | |
| 1952 | 1952 | ATMEL NAND DRIVER |
| 1953 | −M: Josh Wu <josh.wu@atmel.com> | |
| 1953 | +M: Wenyou Yang <wenyou.yang@atmel.com> | |
| 1954 | +M: Josh Wu <rainyfeeling@outlook.com> | |
| 1954 | 1955 | L: linux-mtd@lists.infradead.org |
| 1955 | 1956 | S: Supported |
| 1956 | 1957 | F: drivers/mtd/nand/atmel_nand* |
| @@ -2974,6 +2975,7 @@ F: kernel/cpuset.c · 1017 unmodified lines | ||
| 2974 | 2975 | CONTROL GROUP - MEMORY RESOURCE CONTROLLER (MEMCG) |
| 2975 | 2976 | M: Johannes Weiner <hannes@cmpxchg.org> |
| 2976 | 2977 | M: Michal Hocko <mhocko@kernel.org> |
| 2978 | +M: Vladimir Davydov <vdavydov@virtuozzo.com> | |
| 2977 | 2979 | L: cgroups@vger.kernel.org |
| 2978 | 2980 | L: linux-mm@kvack.org |
| 2979 | 2981 | S: Maintained |
| @@ -6100,6 +6102,7 @@ M: Marc Zyngier <marc.zyngier@arm.com> · 3120 unmodified lines | ||
| 6100 | 6102 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| 6101 | 6103 | L: kvmarm@lists.cs.columbia.edu |
| 6102 | 6104 | W: http://systems.cs.columbia.edu/projects/kvm-arm |
| 6105 | +T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git | |
| 6103 | 6106 | S: Supported |
| 6104 | 6107 | F: arch/arm/include/uapi/asm/kvm* |
| 6105 | 6108 | F: arch/arm/include/asm/kvm* |
| @@ -6366,6 +6369,7 @@ F: arch/*/include/asm/pmem.h · 260 unmodified lines | ||
| 6366 | 6369 | LIGHTNVM PLATFORM SUPPORT |
| 6367 | 6370 | M: Matias Bjorling <mb@lightnvm.io> |
| 6368 | 6371 | W: http://github/OpenChannelSSD |
| 6372 | +L: linux-block@vger.kernel.org | |
| 6369 | 6373 | S: Maintained |
| 6370 | 6374 | F: drivers/lightnvm/ |
| 6371 | 6375 | F: include/linux/lightnvm.h |
| @@ -8284,7 +8288,7 @@ F: include/linux/delayacct.h · 1912 unmodified lines | ||
| 8284 | 8288 | F: kernel/delayacct.c |
| 8285 | 8289 | |
| 8286 | 8290 | PERFORMANCE EVENTS SUBSYSTEM |
| 8287 | −M: Peter Zijlstra <a.p.zijlstra@chello.nl> | |
| 8291 | +M: Peter Zijlstra <peterz@infradead.org> | |
| 8288 | 8292 | M: Ingo Molnar <mingo@redhat.com> |
| 8289 | 8293 | M: Arnaldo Carvalho de Melo <acme@kernel.org> |
| 8290 | 8294 | L: linux-kernel@vger.kernel.org |
| @@ -9425,8 +9429,10 @@ F: include/scsi/sg.h · 1134 unmodified lines | ||
| 9425 | 9429 | |
| 9426 | 9430 | SCSI SUBSYSTEM |
| 9427 | 9431 | M: "James E.J. Bottomley" <JBottomley@odin.com> |
| 9428 | −L: linux-scsi@vger.kernel.org | |
| 9429 | 9432 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git |
| 9433 | +M: "Martin K. Petersen" <martin.petersen@oracle.com> | |
| 9434 | +T: git git://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git | |
| 9435 | +L: linux-scsi@vger.kernel.org | |
| 9430 | 9436 | S: Maintained |
| 9431 | 9437 | F: drivers/scsi/ |
| 9432 | 9438 | F: include/scsi/ |
| @@ -10901,9 +10907,9 @@ S: Maintained · 1468 unmodified lines | ||
| 10901 | 10907 | F: drivers/media/tuners/tua9001* |
| 10902 | 10908 | |
| 10903 | 10909 | TULIP NETWORK DRIVERS |
| 10904 | −M: Grant Grundler <grundler@parisc-linux.org> | |
| 10905 | 10910 | L: netdev@vger.kernel.org |
| 10906 | −S: Maintained | |
| 10911 | +L: linux-parisc@vger.kernel.org | |
| 10912 | +S: Orphan | |
| 10907 | 10913 | F: drivers/net/ethernet/dec/tulip/ |
| 10908 | 10914 | |
| 10909 | 10915 | TUN/TAP driver |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | VERSION = 4 |
| 2 | 2 | PATCHLEVEL = 4 |
| 3 | 3 | SUBLEVEL = 0 |
| 4 | −EXTRAVERSION = -rc2 | |
| 4 | +EXTRAVERSION = -rc5 | |
| 5 | 5 | NAME = Blurry Fish Butt |
| 6 | 6 | |
| 7 | 7 | # *DOCUMENTATION* |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 3 | 3 | # CONFIG_SWAP is not set |
| 4 | 4 | CONFIG_SYSVIPC=y |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 3 | 3 | # CONFIG_SWAP is not set |
| 4 | 4 | CONFIG_SYSVIPC=y |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 3 | 3 | # CONFIG_SWAP is not set |
| 4 | 4 | CONFIG_SYSVIPC=y |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | # CONFIG_LOCALVERSION_AUTO is not set |
| 3 | 3 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 4 | 4 | # CONFIG_SWAP is not set |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | # CONFIG_LOCALVERSION_AUTO is not set |
| 3 | 3 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 4 | 4 | # CONFIG_SWAP is not set |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | # CONFIG_LOCALVERSION_AUTO is not set |
| 3 | 3 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 4 | 4 | # CONFIG_SWAP is not set |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 3 | 3 | # CONFIG_SWAP is not set |
| 4 | 4 | CONFIG_SYSVIPC=y |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | # CONFIG_LOCALVERSION_AUTO is not set |
| 3 | 3 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 4 | 4 | # CONFIG_CROSS_MEMORY_ATTACH is not set |
| @@ -1,4 +1,4 @@ | ||
| 1 | −CONFIG_CROSS_COMPILE="arc-linux-uclibc-" | |
| 1 | +CONFIG_CROSS_COMPILE="arc-linux-" | |
| 2 | 2 | # CONFIG_LOCALVERSION_AUTO is not set |
| 3 | 3 | CONFIG_DEFAULT_HOSTNAME="ARCLinux" |
| 4 | 4 | # CONFIG_CROSS_MEMORY_ATTACH is not set |
| @@ -37,6 +37,9 @@ · 36 unmodified lines | ||
| 37 | 37 | #define ISA_INIT_STATUS_BITS (STATUS_IE_MASK | STATUS_AD_MASK | \ |
| 38 | 38 | (ARCV2_IRQ_DEF_PRIO << 1)) |
| 39 | 39 | |
| 40 | +/* SLEEP needs default irq priority (<=) which can interrupt the doze */ | |
| 41 | +#define ISA_SLEEP_ARG (0x10 | ARCV2_IRQ_DEF_PRIO) | |
| 42 | + | |
| 40 | 43 | #ifndef __ASSEMBLY__ |
| 41 | 44 | |
| 42 | 45 | /* |
| @@ -43,6 +43,8 @@ · 42 unmodified lines | ||
| 43 | 43 | |
| 44 | 44 | #define ISA_INIT_STATUS_BITS STATUS_IE_MASK |
| 45 | 45 | |
| 46 | +#define ISA_SLEEP_ARG 0x3 | |
| 47 | + | |
| 46 | 48 | #ifndef __ASSEMBLY__ |
| 47 | 49 | |
| 48 | 50 | /****************************************************************** |
| @@ -58,8 +58,6 @@ __switch_to(struct task_struct *prev_task, struct task_struct *next_task) · 57 unmodified lines | ||
| 58 | 58 | "st sp, [r24] \n\t" |
| 59 | 59 | #endif |
| 60 | 60 | |
| 61 | − "sync \n\t" | |
| 62 | − | |
| 63 | 61 | /* |
| 64 | 62 | * setup _current_task with incoming tsk. |
| 65 | 63 | * optionally, set r25 to that as well |
| @@ -44,9 +44,6 @@ __switch_to: · 43 unmodified lines | ||
| 44 | 44 | * don't need to do anything special to return it |
| 45 | 45 | */ |
| 46 | 46 | |
| 47 | − /* hardware memory barrier */ | |
| 48 | − sync | |
| 49 | − | |
| 50 | 47 | /* |
| 51 | 48 | * switch to new task, contained in r1 |
| 52 | 49 | * Temp reg r3 is required to get the ptr to store val |
| @@ -44,11 +44,10 @@ SYSCALL_DEFINE0(arc_gettls) · 43 unmodified lines | ||
| 44 | 44 | void arch_cpu_idle(void) |
| 45 | 45 | { |
| 46 | 46 | /* sleep, but enable all interrupts before committing */ |
| 47 | − if (is_isa_arcompact()) { | |
| 48 | − __asm__("sleep 0x3"); | |
| 49 | − } else { | |
| 50 | − __asm__("sleep 0x10"); | |
| 51 | − } | |
| 47 | + __asm__ __volatile__( | |
| 48 | + "sleep %0 \n" | |
| 49 | + : | |
| 50 | + :"I"(ISA_SLEEP_ARG)); /* can't be "r" has to be embedded const */ | |
| 52 | 51 | } |
| 53 | 52 | |
| 54 | 53 | asmlinkage void ret_from_fork(void); |
| @@ -986,42 +986,13 @@ int arc_unwind(struct unwind_frame_info *frame) · 985 unmodified lines | ||
| 986 | 986 | (const u8 *)(fde + |
| 987 | 987 | 1) + |
| 988 | 988 | *fde, ptrType); |
| 989 | − if (pc >= endLoc) | |
| 989 | + if (pc >= endLoc) { | |
| 990 | 990 | fde = NULL; |
| 991 | − } else | |
| 992 | − fde = NULL; | |
| 993 | − } | |
| 994 | − if (fde == NULL) { | |
| 995 | − for (fde = table->address, tableSize = table->size; | |
| 996 | − cie = NULL, tableSize > sizeof(*fde) | |
| 997 | − && tableSize - sizeof(*fde) >= *fde; | |
| 998 | − tableSize -= sizeof(*fde) + *fde, | |
| 999 | − fde += 1 + *fde / sizeof(*fde)) { | |
| 1000 | − cie = cie_for_fde(fde, table); | |
| 1001 | − if (cie == &bad_cie) { | |
| 1002 | 991 | cie = NULL; |
| 1003 | − break; | |
| 1004 | 992 | } |
| 1005 | − if (cie == NULL | |
| 1006 | − || cie == ¬_fde | |
| 1007 | − || (ptrType = fde_pointer_type(cie)) < 0) | |
| 1008 | − continue; | |
| 1009 | − ptr = (const u8 *)(fde + 2); | |
| 1010 | − startLoc = read_pointer(&ptr, | |
| 1011 | − (const u8 *)(fde + 1) + | |
| 1012 | − *fde, ptrType); | |
| 1013 | − if (!startLoc) | |
| 1014 | − continue; | |
| 1015 | − if (!(ptrType & DW_EH_PE_indirect)) | |
| 1016 | − ptrType &= | |
| 1017 | − DW_EH_PE_FORM | DW_EH_PE_signed; | |
| 1018 | − endLoc = | |
| 1019 | − startLoc + read_pointer(&ptr, | |
| 1020 | − (const u8 *)(fde + | |
| 1021 | − 1) + | |
| 1022 | − *fde, ptrType); | |
| 1023 | − if (pc >= startLoc && pc < endLoc) | |
| 1024 | − break; | |
| 993 | + } else { | |
| 994 | + fde = NULL; | |
| 995 | + cie = NULL; | |
| 1025 | 996 | } |
| 1026 | 997 | } |
| 1027 | 998 | } |
| @@ -619,10 +619,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long vaddr_unaligned, · 618 unmodified lines | ||
| 619 | 619 | |
| 620 | 620 | int dirty = !test_and_set_bit(PG_dc_clean, &page->flags); |
| 621 | 621 | if (dirty) { |
| 622 | − /* wback + inv dcache lines */ | |
| 622 | + /* wback + inv dcache lines (K-mapping) */ | |
| 623 | 623 | __flush_dcache_page(paddr, paddr); |
| 624 | 624 | |
| 625 | − /* invalidate any existing icache lines */ | |
| 625 | + /* invalidate any existing icache lines (U-mapping) */ | |
| 626 | 626 | if (vma->vm_flags & VM_EXEC) |
| 627 | 627 | __inv_icache_page(paddr, vaddr); |
| 628 | 628 | } |
| @@ -76,6 +76,8 @@ config ARM · 75 unmodified lines | ||
| 76 | 76 | select IRQ_FORCED_THREADING |
| 77 | 77 | select MODULES_USE_ELF_REL |
| 78 | 78 | select NO_BOOTMEM |
| 79 | + select OF_EARLY_FLATTREE if OF | |
| 80 | + select OF_RESERVED_MEM if OF | |
| 79 | 81 | select OLD_SIGACTION |
| 80 | 82 | select OLD_SIGSUSPEND3 |
| 81 | 83 | select PERF_USE_VMALLOC |
| @@ -1822,8 +1824,6 @@ config USE_OF · 1740 unmodified lines | ||
| 1822 | 1824 | bool "Flattened Device Tree support" |
| 1823 | 1825 | select IRQ_DOMAIN |
| 1824 | 1826 | select OF |
| 1825 | − select OF_EARLY_FLATTREE | |
| 1826 | − select OF_RESERVED_MEM | |
| 1827 | 1827 | help |
| 1828 | 1828 | Include support for flattened device tree machine descriptions. |
| 1829 | 1829 | |
| @@ -74,7 +74,7 @@ · 73 unmodified lines | ||
| 74 | 74 | reg = <0x48240200 0x100>; |
| 75 | 75 | interrupts = <GIC_PPI 11 IRQ_TYPE_LEVEL_HIGH>; |
| 76 | 76 | interrupt-parent = <&gic>; |
| 77 | − clocks = <&dpll_mpu_m2_ck>; | |
| 77 | + clocks = <&mpu_periphclk>; | |
| 78 | 78 | }; |
| 79 | 79 | |
| 80 | 80 | local_timer: timer@48240600 { |
| @@ -82,7 +82,7 @@ · 1 unmodified line | ||
| 82 | 82 | reg = <0x48240600 0x100>; |
| 83 | 83 | interrupts = <GIC_PPI 13 IRQ_TYPE_LEVEL_HIGH>; |
| 84 | 84 | interrupt-parent = <&gic>; |
| 85 | − clocks = <&dpll_mpu_m2_ck>; | |
| 85 | + clocks = <&mpu_periphclk>; | |
| 86 | 86 | }; |
| 87 | 87 | |
| 88 | 88 | l2-cache-controller@48242000 { |
| @@ -259,6 +259,14 @@ · 258 unmodified lines | ||
| 259 | 259 | ti,invert-autoidle-bit; |
| 260 | 260 | }; |
| 261 | 261 | |
| 262 | + mpu_periphclk: mpu_periphclk { | |
| 263 | + #clock-cells = <0>; | |
| 264 | + compatible = "fixed-factor-clock"; | |
| 265 | + clocks = <&dpll_mpu_m2_ck>; | |
| 266 | + clock-mult = <1>; | |
| 267 | + clock-div = <2>; | |
| 268 | + }; | |
| 269 | + | |
| 262 | 270 | dpll_ddr_ck: dpll_ddr_ck { |
| 263 | 271 | #clock-cells = <0>; |
| 264 | 272 | compatible = "ti,am3-dpll-clock"; |
| @@ -604,6 +604,7 @@ · 603 unmodified lines | ||
| 604 | 604 | reg = <0x6f>; |
| 605 | 605 | interrupts-extended = <&crossbar_mpu GIC_SPI 2 IRQ_TYPE_EDGE_RISING>, |
| 606 | 606 | <&dra7_pmx_core 0x424>; |
| 607 | + interrupt-names = "irq", "wakeup"; | |
| 607 | 608 | |
| 608 | 609 | pinctrl-names = "default"; |
| 609 | 610 | pinctrl-0 = <&mcp79410_pins_default>; |
| @@ -155,21 +155,21 @@ · 154 unmodified lines | ||
| 155 | 155 | label = "keyswitch_in"; |
| 156 | 156 | gpios = <&pioB 1 GPIO_ACTIVE_HIGH>; |
| 157 | 157 | linux,code = <28>; |
| 158 | − gpio-key,wakeup; | |
| 158 | + wakeup-source; | |
| 159 | 159 | }; |
| 160 | 160 | |
| 161 | 161 | error_in { |
| 162 | 162 | label = "error_in"; |
| 163 | 163 | gpios = <&pioB 2 GPIO_ACTIVE_HIGH>; |
| 164 | 164 | linux,code = <29>; |
| 165 | − gpio-key,wakeup; | |
| 165 | + wakeup-source; | |
| 166 | 166 | }; |
| 167 | 167 | |
| 168 | 168 | btn { |
| 169 | 169 | label = "btn"; |
| 170 | 170 | gpios = <&pioC 23 GPIO_ACTIVE_HIGH>; |
| 171 | 171 | linux,code = <31>; |
| 172 | − gpio-key,wakeup; | |
| 172 | + wakeup-source; | |
| 173 | 173 | }; |
| 174 | 174 | }; |
| 175 | 175 | }; |
| @@ -498,6 +498,7 @@ · 497 unmodified lines | ||
| 498 | 498 | reg = <0x70000 0x4000>; |
| 499 | 499 | interrupts-extended = <&mpic 8>; |
| 500 | 500 | clocks = <&gateclk 4>; |
| 501 | + tx-csum-limit = <9800>; | |
| 501 | 502 | status = "disabled"; |
| 502 | 503 | }; |
| 503 | 504 | |
| @@ -159,7 +159,7 @@ · 158 unmodified lines | ||
| 159 | 159 | label = "Button"; |
| 160 | 160 | gpios = <&pioC 4 GPIO_ACTIVE_LOW>; |
| 161 | 161 | linux,code = <0x103>; |
| 162 | − gpio-key,wakeup; | |
| 162 | + wakeup-source; | |
| 163 | 163 | }; |
| 164 | 164 | }; |
| 165 | 165 | }; |
| @@ -24,15 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | }; |
| 25 | 25 | |
| 26 | 26 | clocks { |
| 27 | − #address-cells = <1>; | |
| 28 | − #size-cells = <1>; | |
| 29 | − ranges; | |
| 30 | − | |
| 31 | − main_clock: clock@0 { | |
| 32 | − compatible = "atmel,osc", "fixed-clock"; | |
| 33 | − clock-frequency = <18432000>; | |
| 34 | − }; | |
| 35 | − | |
| 36 | 27 | main_xtal { |
| 37 | 28 | clock-frequency = <18432000>; |
| 38 | 29 | }; |
| @@ -94,14 +85,14 @@ · 55 unmodified lines | ||
| 94 | 85 | label = "PB_RST"; |
| 95 | 86 | gpios = <&pioB 30 GPIO_ACTIVE_HIGH>; |
| 96 | 87 | linux,code = <0x100>; |
| 97 | − gpio-key,wakeup; | |
| 88 | + wakeup-source; | |
| 98 | 89 | }; |
| 99 | 90 | |
| 100 | 91 | user { |
| 101 | 92 | label = "PB_USER"; |
| 102 | 93 | gpios = <&pioB 31 GPIO_ACTIVE_HIGH>; |
| 103 | 94 | linux,code = <0x101>; |
| 104 | − gpio-key,wakeup; | |
| 95 | + wakeup-source; | |
| 105 | 96 | }; |
| 106 | 97 | }; |
| 107 | 98 | |
| @@ -171,21 +171,21 @@ · 170 unmodified lines | ||
| 171 | 171 | label = "PB_PROG"; |
| 172 | 172 | gpios = <&pioE 27 GPIO_ACTIVE_LOW>; |
| 173 | 173 | linux,code = <0x102>; |
| 174 | − gpio-key,wakeup; | |
| 174 | + wakeup-source; | |
| 175 | 175 | }; |
| 176 | 176 | |
| 177 | 177 | reset { |
| 178 | 178 | label = "PB_RST"; |
| 179 | 179 | gpios = <&pioE 29 GPIO_ACTIVE_LOW>; |
| 180 | 180 | linux,code = <0x100>; |
| 181 | − gpio-key,wakeup; | |
| 181 | + wakeup-source; | |
| 182 | 182 | }; |
| 183 | 183 | |
| 184 | 184 | user { |
| 185 | 185 | label = "PB_USER"; |
| 186 | 186 | gpios = <&pioE 31 GPIO_ACTIVE_HIGH>; |
| 187 | 187 | linux,code = <0x101>; |
| 188 | − gpio-key,wakeup; | |
| 188 | + wakeup-source; | |
| 189 | 189 | }; |
| 190 | 190 | }; |
| 191 | 191 | |
| @@ -98,14 +98,14 @@ · 97 unmodified lines | ||
| 98 | 98 | label = "PB_PROG"; |
| 99 | 99 | gpios = <&pioC 17 GPIO_ACTIVE_LOW>; |
| 100 | 100 | linux,code = <0x102>; |
| 101 | − gpio-key,wakeup; | |
| 101 | + wakeup-source; | |
| 102 | 102 | }; |
| 103 | 103 | |
| 104 | 104 | reset { |
| 105 | 105 | label = "PB_RST"; |
| 106 | 106 | gpios = <&pioC 16 GPIO_ACTIVE_LOW>; |
| 107 | 107 | linux,code = <0x100>; |
| 108 | − gpio-key,wakeup; | |
| 108 | + wakeup-source; | |
| 109 | 109 | }; |
| 110 | 110 | }; |
| 111 | 111 | |
| @@ -183,7 +183,7 @@ · 182 unmodified lines | ||
| 183 | 183 | label = "user_pb"; |
| 184 | 184 | gpios = <&pioB 10 GPIO_ACTIVE_LOW>; |
| 185 | 185 | linux,code = <28>; |
| 186 | − gpio-key,wakeup; | |
| 186 | + wakeup-source; | |
| 187 | 187 | }; |
| 188 | 188 | }; |
| 189 | 189 | |
| @@ -45,6 +45,7 @@ · 44 unmodified lines | ||
| 45 | 45 | /dts-v1/; |
| 46 | 46 | #include "sama5d2.dtsi" |
| 47 | 47 | #include "sama5d2-pinfunc.h" |
| 48 | +#include <dt-bindings/mfd/atmel-flexcom.h> | |
| 48 | 49 | |
| 49 | 50 | / { |
| 50 | 51 | model = "Atmel SAMA5D2 Xplained"; |
| @@ -59,15 +60,6 @@ · 8 unmodified lines | ||
| 59 | 60 | }; |
| 60 | 61 | |
| 61 | 62 | clocks { |
| 62 | − #address-cells = <1>; | |
| 63 | − #size-cells = <1>; | |
| 64 | − ranges; | |
| 65 | − | |
| 66 | − main_clock: clock@0 { | |
| 67 | − compatible = "atmel,osc", "fixed-clock"; | |
| 68 | − clock-frequency = <12000000>; | |
| 69 | − }; | |
| 70 | − | |
| 71 | 63 | slow_xtal { |
| 72 | 64 | clock-frequency = <32768>; |
| 73 | 65 | }; |
| @@ -91,6 +83,22 @@ · 17 unmodified lines | ||
| 91 | 83 | status = "okay"; |
| 92 | 84 | }; |
| 93 | 85 | |
| 86 | + sdmmc0: sdio-host@a0000000 { | |
| 87 | + bus-width = <8>; | |
| 88 | + pinctrl-names = "default"; | |
| 89 | + pinctrl-0 = <&pinctrl_sdmmc0_default>; | |
| 90 | + non-removable; | |
| 91 | + mmc-ddr-1_8v; | |
| 92 | + status = "okay"; | |
| 93 | + }; | |
| 94 | + | |
| 95 | + sdmmc1: sdio-host@b0000000 { | |
| 96 | + bus-width = <4>; | |
| 97 | + pinctrl-names = "default"; | |
| 98 | + pinctrl-0 = <&pinctrl_sdmmc1_default>; | |
| 99 | + status = "okay"; /* conflict with qspi0 */ | |
| 100 | + }; | |
| 101 | + | |
| 94 | 102 | apb { |
| 95 | 103 | spi0: spi@f8000000 { |
| 96 | 104 | pinctrl-names = "default"; |
| @@ -176,17 +184,55 @@ · 79 unmodified lines | ||
| 176 | 184 | regulator-name = "VDD_SDHC_1V8"; |
| 177 | 185 | regulator-min-microvolt = <1800000>; |
| 178 | 186 | regulator-max-microvolt = <1800000>; |
| 187 | + regulator-always-on; | |
| 179 | 188 | }; |
| 180 | 189 | }; |
| 181 | 190 | }; |
| 182 | 191 | }; |
| 183 | 192 | |
| 193 | + flx0: flexcom@f8034000 { | |
| 194 | + atmel,flexcom-mode = <ATMEL_FLEXCOM_MODE_USART>; | |
| 195 | + status = "disabled"; /* conflict with ISC_D2 & ISC_D3 data pins */ | |
| 196 | + | |
| 197 | + uart5: serial@200 { | |
| 198 | + compatible = "atmel,at91sam9260-usart"; | |
| 199 | + reg = <0x200 0x200>; | |
| 200 | + interrupts = <19 IRQ_TYPE_LEVEL_HIGH 7>; | |
| 201 | + clocks = <&flx0_clk>; | |
| 202 | + clock-names = "usart"; | |
| 203 | + pinctrl-names = "default"; | |
| 204 | + pinctrl-0 = <&pinctrl_flx0_default>; | |
| 205 | + atmel,fifo-size = <32>; | |
| 206 | + status = "okay"; | |
| 207 | + }; | |
| 208 | + }; | |
| 209 | + | |
| 184 | 210 | uart3: serial@fc008000 { |
| 185 | 211 | pinctrl-names = "default"; |
| 186 | 212 | pinctrl-0 = <&pinctrl_uart3_default>; |
| 187 | 213 | status = "okay"; |
| 188 | 214 | }; |
| 189 | 215 | |
| 216 | + flx4: flexcom@fc018000 { | |
| 217 | + atmel,flexcom-mode = <ATMEL_FLEXCOM_MODE_TWI>; | |
| 218 | + status = "okay"; | |
| 219 | + | |
| 220 | + i2c2: i2c@600 { | |
| 221 | + compatible = "atmel,sama5d2-i2c"; | |
| 222 | + reg = <0x600 0x200>; | |
| 223 | + interrupts = <23 IRQ_TYPE_LEVEL_HIGH 7>; | |
| 224 | + dmas = <0>, <0>; | |
| 225 | + dma-names = "tx", "rx"; | |
| 226 | + #address-cells = <1>; | |
| 227 | + #size-cells = <0>; | |
| 228 | + clocks = <&flx4_clk>; | |
| 229 | + pinctrl-names = "default"; | |
| 230 | + pinctrl-0 = <&pinctrl_flx4_default>; | |
| 231 | + atmel,fifo-size = <16>; | |
| 232 | + status = "okay"; | |
| 233 | + }; | |
| 234 | + }; | |
| 235 | + | |
| 190 | 236 | i2c1: i2c@fc028000 { |
| 191 | 237 | dmas = <0>, <0>; |
| 192 | 238 | pinctrl-names = "default"; |
| @@ -201,6 +247,18 @@ · 8 unmodified lines | ||
| 201 | 247 | }; |
| 202 | 248 | |
| 203 | 249 | pinctrl@fc038000 { |
| 250 | + pinctrl_flx0_default: flx0_default { | |
| 251 | + pinmux = <PIN_PB28__FLEXCOM0_IO0>, | |
| 252 | + <PIN_PB29__FLEXCOM0_IO1>; | |
| 253 | + bias-disable; | |
| 254 | + }; | |
| 255 | + | |
| 256 | + pinctrl_flx4_default: flx4_default { | |
| 257 | + pinmux = <PIN_PD12__FLEXCOM4_IO0>, | |
| 258 | + <PIN_PD13__FLEXCOM4_IO1>; | |
| 259 | + bias-disable; | |
| 260 | + }; | |
| 261 | + | |
| 204 | 262 | pinctrl_i2c0_default: i2c0_default { |
| 205 | 263 | pinmux = <PIN_PD21__TWD0>, |
| 206 | 264 | <PIN_PD22__TWCK0>; |
| @@ -227,6 +285,46 @@ · 20 unmodified lines | ||
| 227 | 285 | bias-disable; |
| 228 | 286 | }; |
| 229 | 287 | |
| 288 | + pinctrl_sdmmc0_default: sdmmc0_default { | |
| 289 | + cmd_data { | |
| 290 | + pinmux = <PIN_PA1__SDMMC0_CMD>, | |
| 291 | + <PIN_PA2__SDMMC0_DAT0>, | |
| 292 | + <PIN_PA3__SDMMC0_DAT1>, | |
| 293 | + <PIN_PA4__SDMMC0_DAT2>, | |
| 294 | + <PIN_PA5__SDMMC0_DAT3>, | |
| 295 | + <PIN_PA6__SDMMC0_DAT4>, | |
| 296 | + <PIN_PA7__SDMMC0_DAT5>, | |
| 297 | + <PIN_PA8__SDMMC0_DAT6>, | |
| 298 | + <PIN_PA9__SDMMC0_DAT7>; | |
| 299 | + bias-pull-up; | |
| 300 | + }; | |
| 301 | + | |
| 302 | + ck_cd_rstn_vddsel { | |
| 303 | + pinmux = <PIN_PA0__SDMMC0_CK>, | |
| 304 | + <PIN_PA10__SDMMC0_RSTN>, | |
| 305 | + <PIN_PA11__SDMMC0_VDDSEL>, | |
| 306 | + <PIN_PA13__SDMMC0_CD>; | |
| 307 | + bias-disable; | |
| 308 | + }; | |
| 309 | + }; | |
| 310 | + | |
| 311 | + pinctrl_sdmmc1_default: sdmmc1_default { | |
| 312 | + cmd_data { | |
| 313 | + pinmux = <PIN_PA28__SDMMC1_CMD>, | |
| 314 | + <PIN_PA18__SDMMC1_DAT0>, | |
| 315 | + <PIN_PA19__SDMMC1_DAT1>, | |
| 316 | + <PIN_PA20__SDMMC1_DAT2>, | |
| 317 | + <PIN_PA21__SDMMC1_DAT3>; | |
| 318 | + bias-pull-up; | |
| 319 | + }; | |
| 320 | + | |
| 321 | + conf-ck_cd { | |
| 322 | + pinmux = <PIN_PA22__SDMMC1_CK>, | |
| 323 | + <PIN_PA30__SDMMC1_CD>; | |
| 324 | + bias-disable; | |
| 325 | + }; | |
| 326 | + }; | |
| 327 | + | |
| 230 | 328 | pinctrl_spi0_default: spi0_default { |
| 231 | 329 | pinmux = <PIN_PA14__SPI0_SPCK>, |
| 232 | 330 | <PIN_PA15__SPI0_MOSI>, |
| @@ -315,7 +315,7 @@ · 314 unmodified lines | ||
| 315 | 315 | label = "PB_USER"; |
| 316 | 316 | gpios = <&pioE 29 GPIO_ACTIVE_LOW>; |
| 317 | 317 | linux,code = <0x104>; |
| 318 | − gpio-key,wakeup; | |
| 318 | + wakeup-source; | |
| 319 | 319 | }; |
| 320 | 320 | }; |
| 321 | 321 | |
| @@ -50,7 +50,6 @@ · 49 unmodified lines | ||
| 50 | 50 | compatible = "atmel,sama5d4-xplained", "atmel,sama5d4", "atmel,sama5"; |
| 51 | 51 | |
| 52 | 52 | chosen { |
| 53 | − bootargs = "ignore_loglevel earlyprintk"; | |
| 54 | 53 | stdout-path = "serial0:115200n8"; |
| 55 | 54 | }; |
| 56 | 55 | |
| @@ -59,15 +58,6 @@ · 2 unmodified lines | ||
| 59 | 58 | }; |
| 60 | 59 | |
| 61 | 60 | clocks { |
| 62 | − #address-cells = <1>; | |
| 63 | − #size-cells = <1>; | |
| 64 | − ranges; | |
| 65 | − | |
| 66 | − main_clock: clock@0 { | |
| 67 | − compatible = "atmel,osc", "fixed-clock"; | |
| 68 | − clock-frequency = <12000000>; | |
| 69 | − }; | |
| 70 | − | |
| 71 | 61 | slow_xtal { |
| 72 | 62 | clock-frequency = <32768>; |
| 73 | 63 | }; |
| @@ -235,7 +225,7 @@ · 161 unmodified lines | ||
| 235 | 225 | label = "pb_user1"; |
| 236 | 226 | gpios = <&pioE 8 GPIO_ACTIVE_HIGH>; |
| 237 | 227 | linux,code = <0x100>; |
| 238 | − gpio-key,wakeup; | |
| 228 | + wakeup-source; | |
| 239 | 229 | }; |
| 240 | 230 | }; |
| 241 | 231 | |
| @@ -50,7 +50,6 @@ · 49 unmodified lines | ||
| 50 | 50 | compatible = "atmel,sama5d4ek", "atmel,sama5d4", "atmel,sama5"; |
| 51 | 51 | |
| 52 | 52 | chosen { |
| 53 | − bootargs = "ignore_loglevel earlyprintk"; | |
| 54 | 53 | stdout-path = "serial0:115200n8"; |
| 55 | 54 | }; |
| 56 | 55 | |
| @@ -59,15 +58,6 @@ · 2 unmodified lines | ||
| 59 | 58 | }; |
| 60 | 59 | |
| 61 | 60 | clocks { |
| 62 | − #address-cells = <1>; | |
| 63 | − #size-cells = <1>; | |
| 64 | − ranges; | |
| 65 | − | |
| 66 | − main_clock: clock@0 { | |
| 67 | − compatible = "atmel,osc", "fixed-clock"; | |
| 68 | − clock-frequency = <12000000>; | |
| 69 | − }; | |
| 70 | − | |
| 71 | 61 | slow_xtal { |
| 72 | 62 | clock-frequency = <32768>; |
| 73 | 63 | }; |
| @@ -304,7 +294,7 @@ · 230 unmodified lines | ||
| 304 | 294 | label = "pb_user1"; |
| 305 | 295 | gpios = <&pioE 13 GPIO_ACTIVE_HIGH>; |
| 306 | 296 | linux,code = <0x100>; |
| 307 | − gpio-key,wakeup; | |
| 297 | + wakeup-source; | |
| 308 | 298 | }; |
| 309 | 299 | }; |
| 310 | 300 | |
| @@ -21,15 +21,6 @@ · 20 unmodified lines | ||
| 21 | 21 | }; |
| 22 | 22 | |
| 23 | 23 | clocks { |
| 24 | − #address-cells = <1>; | |
| 25 | − #size-cells = <1>; | |
| 26 | − ranges; | |
| 27 | − | |
| 28 | − main_clock: clock@0 { | |
| 29 | − compatible = "atmel,osc", "fixed-clock"; | |
| 30 | − clock-frequency = <18432000>; | |
| 31 | − }; | |
| 32 | − | |
| 33 | 24 | slow_xtal { |
| 34 | 25 | clock-frequency = <32768>; |
| 35 | 26 | }; |
| @@ -22,15 +22,6 @@ · 21 unmodified lines | ||
| 22 | 22 | }; |
| 23 | 23 | |
| 24 | 24 | clocks { |
| 25 | − #address-cells = <1>; | |
| 26 | − #size-cells = <1>; | |
| 27 | − ranges; | |
| 28 | − | |
| 29 | − main_clock: clock@0 { | |
| 30 | − compatible = "atmel,osc", "fixed-clock"; | |
| 31 | − clock-frequency = <18432000>; | |
| 32 | − }; | |
| 33 | − | |
| 34 | 25 | slow_xtal { |
| 35 | 26 | clock-frequency = <32768>; |
| 36 | 27 | }; |
| @@ -149,7 +140,7 @@ · 112 unmodified lines | ||
| 149 | 140 | ti,debounce-tol = /bits/ 16 <65535>; |
| 150 | 141 | ti,debounce-max = /bits/ 16 <1>; |
| 151 | 142 | |
| 152 | − linux,wakeup; | |
| 143 | + wakeup-source; | |
| 153 | 144 | }; |
| 154 | 145 | }; |
| 155 | 146 | |
| @@ -193,28 +184,28 @@ · 37 unmodified lines | ||
| 193 | 184 | label = "button_0"; |
| 194 | 185 | gpios = <&pioA 27 GPIO_ACTIVE_LOW>; |
| 195 | 186 | linux,code = <256>; |
| 196 | − gpio-key,wakeup; | |
| 187 | + wakeup-source; | |
| 197 | 188 | }; |
| 198 | 189 | |
| 199 | 190 | button_1 { |
| 200 | 191 | label = "button_1"; |
| 201 | 192 | gpios = <&pioA 26 GPIO_ACTIVE_LOW>; |
| 202 | 193 | linux,code = <257>; |
| 203 | − gpio-key,wakeup; | |
| 194 | + wakeup-source; | |
| 204 | 195 | }; |
| 205 | 196 | |
| 206 | 197 | button_2 { |
| 207 | 198 | label = "button_2"; |
| 208 | 199 | gpios = <&pioA 25 GPIO_ACTIVE_LOW>; |
| 209 | 200 | linux,code = <258>; |
| 210 | − gpio-key,wakeup; | |
| 201 | + wakeup-source; | |
| 211 | 202 | }; |
| 212 | 203 | |
| 213 | 204 | button_3 { |
| 214 | 205 | label = "button_3"; |
| 215 | 206 | gpios = <&pioA 24 GPIO_ACTIVE_LOW>; |
| 216 | 207 | linux,code = <259>; |
| 217 | − gpio-key,wakeup; | |
| 208 | + wakeup-source; | |
| 218 | 209 | }; |
| 219 | 210 | }; |
| 220 | 211 | }; |
| @@ -22,15 +22,6 @@ · 21 unmodified lines | ||
| 22 | 22 | }; |
| 23 | 23 | |
| 24 | 24 | clocks { |
| 25 | − #address-cells = <1>; | |
| 26 | − #size-cells = <1>; | |
| 27 | − ranges; | |
| 28 | − | |
| 29 | − main_clock: clock@0 { | |
| 30 | − compatible = "atmel,osc", "fixed-clock"; | |
| 31 | − clock-frequency = <16367660>; | |
| 32 | − }; | |
| 33 | − | |
| 34 | 25 | slow_xtal { |
| 35 | 26 | clock-frequency = <32768>; |
| 36 | 27 | }; |
| @@ -213,14 +204,14 @@ · 176 unmodified lines | ||
| 213 | 204 | label = "left_click"; |
| 214 | 205 | gpios = <&pioC 5 GPIO_ACTIVE_LOW>; |
| 215 | 206 | linux,code = <272>; |
| 216 | − gpio-key,wakeup; | |
| 207 | + wakeup-source; | |
| 217 | 208 | }; |
| 218 | 209 | |
| 219 | 210 | right_click { |
| 220 | 211 | label = "right_click"; |
| 221 | 212 | gpios = <&pioC 4 GPIO_ACTIVE_LOW>; |
| 222 | 213 | linux,code = <273>; |
| 223 | − gpio-key,wakeup; | |
| 214 | + wakeup-source; | |
| 224 | 215 | }; |
| 225 | 216 | }; |
| 226 | 217 | |
| @@ -19,15 +19,6 @@ · 18 unmodified lines | ||
| 19 | 19 | }; |
| 20 | 20 | |
| 21 | 21 | clocks { |
| 22 | − #address-cells = <1>; | |
| 23 | − #size-cells = <1>; | |
| 24 | − ranges; | |
| 25 | − | |
| 26 | − main_clock: clock@0 { | |
| 27 | − compatible = "atmel,osc", "fixed-clock"; | |
| 28 | − clock-frequency = <18432000>; | |
| 29 | − }; | |
| 30 | − | |
| 31 | 22 | slow_xtal { |
| 32 | 23 | clock-frequency = <32768>; |
| 33 | 24 | }; |
| @@ -206,14 +197,14 @@ · 172 unmodified lines | ||
| 206 | 197 | label = "Button 3"; |
| 207 | 198 | gpios = <&pioA 30 GPIO_ACTIVE_LOW>; |
| 208 | 199 | linux,code = <0x103>; |
| 209 | − gpio-key,wakeup; | |
| 200 | + wakeup-source; | |
| 210 | 201 | }; |
| 211 | 202 | |
| 212 | 203 | btn4 { |
| 213 | 204 | label = "Button 4"; |
| 214 | 205 | gpios = <&pioA 31 GPIO_ACTIVE_LOW>; |
| 215 | 206 | linux,code = <0x104>; |
| 216 | − gpio-key,wakeup; | |
| 207 | + wakeup-source; | |
| 217 | 208 | }; |
| 218 | 209 | }; |
| 219 | 210 | |
| @@ -24,15 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | }; |
| 25 | 25 | |
| 26 | 26 | clocks { |
| 27 | − #address-cells = <1>; | |
| 28 | − #size-cells = <1>; | |
| 29 | − ranges; | |
| 30 | − | |
| 31 | − main_clock: clock@0 { | |
| 32 | − compatible = "atmel,osc", "fixed-clock"; | |
| 33 | − clock-frequency = <12000000>; | |
| 34 | − }; | |
| 35 | − | |
| 36 | 27 | slow_xtal { |
| 37 | 28 | clock-frequency = <32768>; |
| 38 | 29 | }; |
| @@ -323,14 +314,14 @@ · 284 unmodified lines | ||
| 323 | 314 | label = "left_click"; |
| 324 | 315 | gpios = <&pioB 6 GPIO_ACTIVE_LOW>; |
| 325 | 316 | linux,code = <272>; |
| 326 | − gpio-key,wakeup; | |
| 317 | + wakeup-source; | |
| 327 | 318 | }; |
| 328 | 319 | |
| 329 | 320 | right_click { |
| 330 | 321 | label = "right_click"; |
| 331 | 322 | gpios = <&pioB 7 GPIO_ACTIVE_LOW>; |
| 332 | 323 | linux,code = <273>; |
| 333 | − gpio-key,wakeup; | |
| 324 | + wakeup-source; | |
| 334 | 325 | }; |
| 335 | 326 | |
| 336 | 327 | left { |
| @@ -23,15 +23,6 @@ · 22 unmodified lines | ||
| 23 | 23 | }; |
| 24 | 24 | |
| 25 | 25 | clocks { |
| 26 | − #address-cells = <1>; | |
| 27 | − #size-cells = <1>; | |
| 28 | − ranges; | |
| 29 | − | |
| 30 | − main_clock: clock@0 { | |
| 31 | − compatible = "atmel,osc", "fixed-clock"; | |
| 32 | − clock-frequency = <16000000>; | |
| 33 | − }; | |
| 34 | − | |
| 35 | 26 | slow_xtal { |
| 36 | 27 | clock-frequency = <32768>; |
| 37 | 28 | }; |
| @@ -219,7 +210,7 @@ · 181 unmodified lines | ||
| 219 | 210 | label = "Enter"; |
| 220 | 211 | gpios = <&pioB 3 GPIO_ACTIVE_LOW>; |
| 221 | 212 | linux,code = <28>; |
| 222 | − gpio-key,wakeup; | |
| 213 | + wakeup-source; | |
| 223 | 214 | }; |
| 224 | 215 | }; |
| 225 | 216 | |
| @@ -22,15 +22,6 @@ · 21 unmodified lines | ||
| 22 | 22 | }; |
| 23 | 23 | |
| 24 | 24 | clocks { |
| 25 | − #address-cells = <1>; | |
| 26 | − #size-cells = <1>; | |
| 27 | − ranges; | |
| 28 | − | |
| 29 | − main_clock: clock { | |
| 30 | − compatible = "atmel,osc", "fixed-clock"; | |
| 31 | − clock-frequency = <12000000>; | |
| 32 | − }; | |
| 33 | − | |
| 34 | 25 | slow_xtal { |
| 35 | 26 | clock-frequency = <32768>; |
| 36 | 27 | }; |
| @@ -225,14 +216,14 @@ · 188 unmodified lines | ||
| 225 | 216 | label = "right_click"; |
| 226 | 217 | gpios = <&pioB 0 GPIO_ACTIVE_LOW>; |
| 227 | 218 | linux,code = <273>; |
| 228 | − gpio-key,wakeup; | |
| 219 | + wakeup-source; | |
| 229 | 220 | }; |
| 230 | 221 | |
| 231 | 222 | left_click { |
| 232 | 223 | label = "left_click"; |
| 233 | 224 | gpios = <&pioB 1 GPIO_ACTIVE_LOW>; |
| 234 | 225 | linux,code = <272>; |
| 235 | − gpio-key,wakeup; | |
| 226 | + wakeup-source; | |
| 236 | 227 | }; |
| 237 | 228 | }; |
| 238 | 229 | |
| @@ -12,17 +12,6 @@ · 11 unmodified lines | ||
| 12 | 12 | reg = <0x20000000 0x8000000>; |
| 13 | 13 | }; |
| 14 | 14 | |
| 15 | − clocks { | |
| 16 | − #address-cells = <1>; | |
| 17 | − #size-cells = <1>; | |
| 18 | − ranges; | |
| 19 | − | |
| 20 | − main_clock: clock@0 { | |
| 21 | − compatible = "atmel,osc", "fixed-clock"; | |
| 22 | − clock-frequency = <12000000>; | |
| 23 | − }; | |
| 24 | − }; | |
| 25 | − | |
| 26 | 15 | clocks { |
| 27 | 16 | slow_xtal { |
| 28 | 17 | clock-frequency = <32768>; |
| @@ -118,7 +118,8 @@ · 117 unmodified lines | ||
| 118 | 118 | sdhci0: sdhci@ab0000 { |
| 119 | 119 | compatible = "mrvl,pxav3-mmc"; |
| 120 | 120 | reg = <0xab0000 0x200>; |
| 121 | − clocks = <&chip_clk CLKID_SDIO1XIN>; | |
| 121 | + clocks = <&chip_clk CLKID_SDIO1XIN>, <&chip_clk CLKID_SDIO>; | |
| 122 | + clock-names = "io", "core"; | |
| 122 | 123 | interrupts = <GIC_SPI 17 IRQ_TYPE_LEVEL_HIGH>; |
| 123 | 124 | status = "disabled"; |
| 124 | 125 | }; |
| @@ -126,7 +127,8 @@ · 1 unmodified line | ||
| 126 | 127 | sdhci1: sdhci@ab0800 { |
| 127 | 128 | compatible = "mrvl,pxav3-mmc"; |
| 128 | 129 | reg = <0xab0800 0x200>; |
| 129 | − clocks = <&chip_clk CLKID_SDIO1XIN>; | |
| 130 | + clocks = <&chip_clk CLKID_SDIO1XIN>, <&chip_clk CLKID_SDIO>; | |
| 131 | + clock-names = "io", "core"; | |
| 130 | 132 | interrupts = <GIC_SPI 20 IRQ_TYPE_LEVEL_HIGH>; |
| 131 | 133 | status = "disabled"; |
| 132 | 134 | }; |
| @@ -135,7 +137,7 @@ · 2 unmodified lines | ||
| 135 | 137 | compatible = "mrvl,pxav3-mmc"; |
| 136 | 138 | reg = <0xab1000 0x200>; |
| 137 | 139 | interrupts = <GIC_SPI 28 IRQ_TYPE_LEVEL_HIGH>; |
| 138 | − clocks = <&chip_clk CLKID_NFC_ECC>, <&chip_clk CLKID_NFC>; | |
| 140 | + clocks = <&chip_clk CLKID_NFC_ECC>, <&chip_clk CLKID_SDIO>; | |
| 139 | 141 | clock-names = "io", "core"; |
| 140 | 142 | status = "disabled"; |
| 141 | 143 | }; |
| @@ -218,6 +218,7 @@ · 217 unmodified lines | ||
| 218 | 218 | reg = <0x480c8000 0x2000>; |
| 219 | 219 | interrupts = <77>; |
| 220 | 220 | ti,hwmods = "mailbox"; |
| 221 | + #mbox-cells = <1>; | |
| 221 | 222 | ti,mbox-num-users = <4>; |
| 222 | 223 | ti,mbox-num-fifos = <12>; |
| 223 | 224 | mbox_dsp: mbox_dsp { |
| @@ -279,8 +280,11 @@ · 55 unmodified lines | ||
| 279 | 280 | ti,spi-num-cs = <4>; |
| 280 | 281 | ti,hwmods = "mcspi1"; |
| 281 | 282 | dmas = <&edma 16 &edma 17 |
| 282 | − &edma 18 &edma 19>; | |
| 283 | − dma-names = "tx0", "rx0", "tx1", "rx1"; | |
| 283 | + &edma 18 &edma 19 | |
| 284 | + &edma 20 &edma 21 | |
| 285 | + &edma 22 &edma 23>; | |
| 286 | + dma-names = "tx0", "rx0", "tx1", "rx1", | |
| 287 | + "tx2", "rx2", "tx3", "rx3"; | |
| 284 | 288 | }; |
| 285 | 289 | |
| 286 | 290 | mmc1: mmc@48060000 { |
| @@ -1459,8 +1459,8 @@ · 1458 unmodified lines | ||
| 1459 | 1459 | interrupt-names = "tx", "rx"; |
| 1460 | 1460 | dmas = <&sdma_xbar 133>, <&sdma_xbar 132>; |
| 1461 | 1461 | dma-names = "tx", "rx"; |
| 1462 | − clocks = <&mcasp3_ahclkx_mux>; | |
| 1463 | − clock-names = "fck"; | |
| 1462 | + clocks = <&mcasp3_aux_gfclk_mux>, <&mcasp3_ahclkx_mux>; | |
| 1463 | + clock-names = "fck", "ahclkx"; | |
| 1464 | 1464 | status = "disabled"; |
| 1465 | 1465 | }; |
| 1466 | 1466 | |
| @@ -137,7 +137,7 @@ netcp: netcp@26000000 { · 136 unmodified lines | ||
| 137 | 137 | /* NetCP address range */ |
| 138 | 138 | ranges = <0 0x26000000 0x1000000>; |
| 139 | 139 | |
| 140 | − clocks = <&papllclk>, <&clkcpgmac>, <&chipclk12>; | |
| 140 | + clocks = <&clkosr>, <&papllclk>, <&clkcpgmac>, <&chipclk12>; | |
| 141 | 141 | dma-coherent; |
| 142 | 142 | |
| 143 | 143 | ti,navigator-dmas = <&dma_gbe 0>, |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | ||
| 40 | 40 | }; |
| 41 | 41 | poweroff@12100 { |
| 42 | 42 | compatible = "qnap,power-off"; |
| 43 | − reg = <0x12000 0x100>; | |
| 43 | + reg = <0x12100 0x100>; | |
| 44 | 44 | clocks = <&gate_clk 7>; |
| 45 | 45 | }; |
| 46 | 46 | spi@10600 { |
| @@ -86,6 +86,10 @@ · 85 unmodified lines | ||
| 86 | 86 | }; |
| 87 | 87 | }; |
| 88 | 88 | |
| 89 | +&emmc { | |
| 90 | + /delete-property/mmc-hs200-1_8v; | |
| 91 | +}; | |
| 92 | + | |
| 89 | 93 | &gpio_keys { |
| 90 | 94 | pinctrl-0 = <&pwr_key_l &ap_lid_int_l &volum_down_l &volum_up_l>; |
| 91 | 95 | |
| @@ -452,8 +452,10 @@ · 451 unmodified lines | ||
| 452 | 452 | clock-names = "tsadc", "apb_pclk"; |
| 453 | 453 | resets = <&cru SRST_TSADC>; |
| 454 | 454 | reset-names = "tsadc-apb"; |
| 455 | − pinctrl-names = "default"; | |
| 456 | − pinctrl-0 = <&otp_out>; | |
| 455 | + pinctrl-names = "init", "default", "sleep"; | |
| 456 | + pinctrl-0 = <&otp_gpio>; | |
| 457 | + pinctrl-1 = <&otp_out>; | |
| 458 | + pinctrl-2 = <&otp_gpio>; | |
| 457 | 459 | #thermal-sensor-cells = <1>; |
| 458 | 460 | rockchip,hw-tshut-temp = <95000>; |
| 459 | 461 | status = "disabled"; |
| @@ -1395,6 +1397,10 @@ · 935 unmodified lines | ||
| 1395 | 1397 | }; |
| 1396 | 1398 | |
| 1397 | 1399 | tsadc { |
| 1400 | + otp_gpio: otp-gpio { | |
| 1401 | + rockchip,pins = <0 10 RK_FUNC_GPIO &pcfg_pull_none>; | |
| 1402 | + }; | |
| 1403 | + | |
| 1398 | 1404 | otp_out: otp-out { |
| 1399 | 1405 | rockchip,pins = <0 10 RK_FUNC_1 &pcfg_pull_none>; |
| 1400 | 1406 | }; |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | ||
| 49 | 49 | label = "pb_user1"; |
| 50 | 50 | gpios = <&pioE 27 GPIO_ACTIVE_HIGH>; |
| 51 | 51 | linux,code = <0x100>; |
| 52 | − gpio-key,wakeup; | |
| 52 | + wakeup-source; | |
| 53 | 53 | }; |
| 54 | 54 | }; |
| 55 | 55 | }; |
| @@ -1300,7 +1300,7 @@ · 1299 unmodified lines | ||
| 1300 | 1300 | }; |
| 1301 | 1301 | |
| 1302 | 1302 | watchdog@fc068640 { |
| 1303 | − compatible = "atmel,at91sam9260-wdt"; | |
| 1303 | + compatible = "atmel,sama5d4-wdt"; | |
| 1304 | 1304 | reg = <0xfc068640 0x10>; |
| 1305 | 1305 | clocks = <&clk32k>; |
| 1306 | 1306 | status = "disabled"; |
| @@ -115,7 +115,7 @@ · 114 unmodified lines | ||
| 115 | 115 | label = "user_pb"; |
| 116 | 116 | gpios = <&pioB 10 GPIO_ACTIVE_LOW>; |
| 117 | 117 | linux,code = <28>; |
| 118 | − gpio-key,wakeup; | |
| 118 | + wakeup-source; | |
| 119 | 119 | }; |
| 120 | 120 | }; |
| 121 | 121 | |
| @@ -143,7 +143,7 @@ · 142 unmodified lines | ||
| 143 | 143 | label = "user_pb"; |
| 144 | 144 | gpios = <&pioB 10 GPIO_ACTIVE_LOW>; |
| 145 | 145 | linux,code = <28>; |
| 146 | − gpio-key,wakeup; | |
| 146 | + wakeup-source; | |
| 147 | 147 | }; |
| 148 | 148 | }; |
| 149 | 149 | |
| @@ -18,8 +18,3 @@ · 17 unmodified lines | ||
| 18 | 18 | reg = <0x80000000 0x10000000>; |
| 19 | 19 | }; |
| 20 | 20 | }; |
| 21 | − | |
| 22 | −&L2 { | |
| 23 | − arm,data-latency = <2 1 2>; | |
| 24 | − arm,tag-latency = <3 2 3>; | |
| 25 | −}; | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | reg = <0x40006000 0x1000>; |
| 20 | 20 | cache-unified; |
| 21 | 21 | cache-level = <2>; |
| 22 | − arm,data-latency = <1 1 1>; | |
| 22 | + arm,data-latency = <3 3 3>; | |
| 23 | 23 | arm,tag-latency = <2 2 2>; |
| 24 | 24 | }; |
| 25 | 25 | }; |
| @@ -158,7 +158,7 @@ · 157 unmodified lines | ||
| 158 | 158 | interrupts = <67 IRQ_TYPE_LEVEL_HIGH>; |
| 159 | 159 | clocks = <&clks VF610_CLK_DSPI0>; |
| 160 | 160 | clock-names = "dspi"; |
| 161 | − spi-num-chipselects = <5>; | |
| 161 | + spi-num-chipselects = <6>; | |
| 162 | 162 | status = "disabled"; |
| 163 | 163 | }; |
| 164 | 164 | |
| @@ -170,7 +170,7 @@ · 5 unmodified lines | ||
| 170 | 170 | interrupts = <68 IRQ_TYPE_LEVEL_HIGH>; |
| 171 | 171 | clocks = <&clks VF610_CLK_DSPI1>; |
| 172 | 172 | clock-names = "dspi"; |
| 173 | − spi-num-chipselects = <5>; | |
| 173 | + spi-num-chipselects = <4>; | |
| 174 | 174 | status = "disabled"; |
| 175 | 175 | }; |
| 176 | 176 | |
| @@ -178,8 +178,10 @@ · 1 unmodified line | ||
| 178 | 178 | compatible = "fsl,vf610-sai"; |
| 179 | 179 | reg = <0x40031000 0x1000>; |
| 180 | 180 | interrupts = <86 IRQ_TYPE_LEVEL_HIGH>; |
| 181 | − clocks = <&clks VF610_CLK_SAI2>; | |
| 182 | − clock-names = "sai"; | |
| 181 | + clocks = <&clks VF610_CLK_SAI2>, | |
| 182 | + <&clks VF610_CLK_SAI2_DIV>, | |
| 183 | + <&clks 0>, <&clks 0>; | |
| 184 | + clock-names = "bus", "mclk1", "mclk2", "mclk3"; | |
| 183 | 185 | dma-names = "tx", "rx"; |
| 184 | 186 | dmas = <&edma0 0 21>, |
| 185 | 187 | <&edma0 0 20>; |
| @@ -461,6 +463,8 @@ · 275 unmodified lines | ||
| 461 | 463 | clock-names = "adc"; |
| 462 | 464 | #io-channel-cells = <1>; |
| 463 | 465 | status = "disabled"; |
| 466 | + fsl,adck-max-frequency = <30000000>, <40000000>, | |
| 467 | + <20000000>; | |
| 464 | 468 | }; |
| 465 | 469 | |
| 466 | 470 | esdhc0: esdhc@400b1000 { |
| @@ -472,8 +476,6 @@ · 5 unmodified lines | ||
| 472 | 476 | <&clks VF610_CLK_ESDHC0>; |
| 473 | 477 | clock-names = "ipg", "ahb", "per"; |
| 474 | 478 | status = "disabled"; |
| 475 | − fsl,adck-max-frequency = <30000000>, <40000000>, | |
| 476 | − <20000000>; | |
| 477 | 479 | }; |
| 478 | 480 | |
| 479 | 481 | esdhc1: esdhc@400b2000 { |
| @@ -125,7 +125,6 @@ CONFIG_POWER_RESET=y · 124 unmodified lines | ||
| 125 | 125 | # CONFIG_HWMON is not set |
| 126 | 126 | CONFIG_WATCHDOG=y |
| 127 | 127 | CONFIG_AT91SAM9X_WATCHDOG=y |
| 128 | −CONFIG_SSB=m | |
| 129 | 128 | CONFIG_MFD_ATMEL_HLCDC=y |
| 130 | 129 | CONFIG_REGULATOR=y |
| 131 | 130 | CONFIG_REGULATOR_FIXED_VOLTAGE=y |
| @@ -129,7 +129,6 @@ CONFIG_GPIO_SYSFS=y · 128 unmodified lines | ||
| 129 | 129 | CONFIG_POWER_SUPPLY=y |
| 130 | 130 | CONFIG_POWER_RESET=y |
| 131 | 131 | # CONFIG_HWMON is not set |
| 132 | −CONFIG_SSB=m | |
| 133 | 132 | CONFIG_MFD_ATMEL_FLEXCOM=y |
| 134 | 133 | CONFIG_REGULATOR=y |
| 135 | 134 | CONFIG_REGULATOR_FIXED_VOLTAGE=y |
| @@ -21,6 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #ifndef __ASSEMBLY__ |
| 22 | 22 | |
| 23 | 23 | #include <linux/io.h> |
| 24 | +#include <asm/barrier.h> | |
| 24 | 25 | |
| 25 | 26 | #define __ACCESS_CP15(CRn, Op1, CRm, Op2) p15, Op1, %0, CRn, CRm, Op2 |
| 26 | 27 | #define __ACCESS_CP15_64(Op1, CRm) p15, Op1, %Q0, %R0, CRm |
| @@ -40,6 +40,11 @@ extern void arch_trigger_all_cpu_backtrace(bool); · 39 unmodified lines | ||
| 40 | 40 | #define arch_trigger_all_cpu_backtrace(x) arch_trigger_all_cpu_backtrace(x) |
| 41 | 41 | #endif |
| 42 | 42 | |
| 43 | +static inline int nr_legacy_irqs(void) | |
| 44 | +{ | |
| 45 | + return NR_IRQS_LEGACY; | |
| 46 | +} | |
| 47 | + | |
| 43 | 48 | #endif |
| 44 | 49 | |
| 45 | 50 | #endif |
| @@ -19,6 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #ifndef __ARM_KVM_ARM_H__ |
| 20 | 20 | #define __ARM_KVM_ARM_H__ |
| 21 | 21 | |
| 22 | +#include <linux/const.h> | |
| 22 | 23 | #include <linux/types.h> |
| 23 | 24 | |
| 24 | 25 | /* Hyp Configuration Register (HCR) bits */ |
| @@ -132,10 +133,9 @@ · 107 unmodified lines | ||
| 132 | 133 | * space. |
| 133 | 134 | */ |
| 134 | 135 | #define KVM_PHYS_SHIFT (40) |
| 135 | −#define KVM_PHYS_SIZE (1ULL << KVM_PHYS_SHIFT) | |
| 136 | −#define KVM_PHYS_MASK (KVM_PHYS_SIZE - 1ULL) | |
| 137 | −#define PTRS_PER_S2_PGD (1ULL << (KVM_PHYS_SHIFT - 30)) | |
| 138 | −#define S2_PGD_ORDER get_order(PTRS_PER_S2_PGD * sizeof(pgd_t)) | |
| 136 | +#define KVM_PHYS_SIZE (_AC(1, ULL) << KVM_PHYS_SHIFT) | |
| 137 | +#define KVM_PHYS_MASK (KVM_PHYS_SIZE - _AC(1, ULL)) | |
| 138 | +#define PTRS_PER_S2_PGD (_AC(1, ULL) << (KVM_PHYS_SHIFT - 30)) | |
| 139 | 139 | |
| 140 | 140 | /* Virtualization Translation Control Register (VTCR) bits */ |
| 141 | 141 | #define VTCR_SH0 (3 << 12) |
| @@ -162,17 +162,17 @@ · 20 unmodified lines | ||
| 162 | 162 | #define VTTBR_X (5 - KVM_T0SZ) |
| 163 | 163 | #endif |
| 164 | 164 | #define VTTBR_BADDR_SHIFT (VTTBR_X - 1) |
| 165 | −#define VTTBR_BADDR_MASK (((1LLU << (40 - VTTBR_X)) - 1) << VTTBR_BADDR_SHIFT) | |
| 166 | −#define VTTBR_VMID_SHIFT (48LLU) | |
| 167 | −#define VTTBR_VMID_MASK (0xffLLU << VTTBR_VMID_SHIFT) | |
| 165 | +#define VTTBR_BADDR_MASK (((_AC(1, ULL) << (40 - VTTBR_X)) - 1) << VTTBR_BADDR_SHIFT) | |
| 166 | +#define VTTBR_VMID_SHIFT _AC(48, ULL) | |
| 167 | +#define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) | |
| 168 | 168 | |
| 169 | 169 | /* Hyp Syndrome Register (HSR) bits */ |
| 170 | 170 | #define HSR_EC_SHIFT (26) |
| 171 | −#define HSR_EC (0x3fU << HSR_EC_SHIFT) | |
| 172 | −#define HSR_IL (1U << 25) | |
| 171 | +#define HSR_EC (_AC(0x3f, UL) << HSR_EC_SHIFT) | |
| 172 | +#define HSR_IL (_AC(1, UL) << 25) | |
| 173 | 173 | #define HSR_ISS (HSR_IL - 1) |
| 174 | 174 | #define HSR_ISV_SHIFT (24) |
| 175 | −#define HSR_ISV (1U << HSR_ISV_SHIFT) | |
| 175 | +#define HSR_ISV (_AC(1, UL) << HSR_ISV_SHIFT) | |
| 176 | 176 | #define HSR_SRT_SHIFT (16) |
| 177 | 177 | #define HSR_SRT_MASK (0xf << HSR_SRT_SHIFT) |
| 178 | 178 | #define HSR_FSC (0x3f) |
| @@ -180,9 +180,9 @@ · 1 unmodified line | ||
| 180 | 180 | #define HSR_SSE (1 << 21) |
| 181 | 181 | #define HSR_WNR (1 << 6) |
| 182 | 182 | #define HSR_CV_SHIFT (24) |
| 183 | −#define HSR_CV (1U << HSR_CV_SHIFT) | |
| 183 | +#define HSR_CV (_AC(1, UL) << HSR_CV_SHIFT) | |
| 184 | 184 | #define HSR_COND_SHIFT (20) |
| 185 | −#define HSR_COND (0xfU << HSR_COND_SHIFT) | |
| 185 | +#define HSR_COND (_AC(0xf, UL) << HSR_COND_SHIFT) | |
| 186 | 186 | |
| 187 | 187 | #define FSC_FAULT (0x04) |
| 188 | 188 | #define FSC_ACCESS (0x08) |
| @@ -210,13 +210,13 @@ · 21 unmodified lines | ||
| 210 | 210 | #define HSR_EC_DABT (0x24) |
| 211 | 211 | #define HSR_EC_DABT_HYP (0x25) |
| 212 | 212 | |
| 213 | −#define HSR_WFI_IS_WFE (1U << 0) | |
| 213 | +#define HSR_WFI_IS_WFE (_AC(1, UL) << 0) | |
| 214 | 214 | |
| 215 | −#define HSR_HVC_IMM_MASK ((1UL << 16) - 1) | |
| 215 | +#define HSR_HVC_IMM_MASK ((_AC(1, UL) << 16) - 1) | |
| 216 | 216 | |
| 217 | −#define HSR_DABT_S1PTW (1U << 7) | |
| 218 | −#define HSR_DABT_CM (1U << 8) | |
| 219 | −#define HSR_DABT_EA (1U << 9) | |
| 217 | +#define HSR_DABT_S1PTW (_AC(1, UL) << 7) | |
| 218 | +#define HSR_DABT_CM (_AC(1, UL) << 8) | |
| 219 | +#define HSR_DABT_EA (_AC(1, UL) << 9) | |
| 220 | 220 | |
| 221 | 221 | #define kvm_arm_exception_type \ |
| 222 | 222 | {0, "RESET" }, \ |
| @@ -28,6 +28,18 @@ · 27 unmodified lines | ||
| 28 | 28 | unsigned long *vcpu_reg(struct kvm_vcpu *vcpu, u8 reg_num); |
| 29 | 29 | unsigned long *vcpu_spsr(struct kvm_vcpu *vcpu); |
| 30 | 30 | |
| 31 | +static inline unsigned long vcpu_get_reg(struct kvm_vcpu *vcpu, | |
| 32 | + u8 reg_num) | |
| 33 | +{ | |
| 34 | + return *vcpu_reg(vcpu, reg_num); | |
| 35 | +} | |
| 36 | + | |
| 37 | +static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, | |
| 38 | + unsigned long val) | |
| 39 | +{ | |
| 40 | + *vcpu_reg(vcpu, reg_num) = val; | |
| 41 | +} | |
| 42 | + | |
| 31 | 43 | bool kvm_condition_valid(struct kvm_vcpu *vcpu); |
| 32 | 44 | void kvm_skip_instr(struct kvm_vcpu *vcpu, bool is_wide_instr); |
| 33 | 45 | void kvm_inject_undefined(struct kvm_vcpu *vcpu); |
| @@ -150,6 +150,12 @@ struct kvm_vcpu_stat { · 149 unmodified lines | ||
| 150 | 150 | u32 halt_successful_poll; |
| 151 | 151 | u32 halt_attempted_poll; |
| 152 | 152 | u32 halt_wakeup; |
| 153 | + u32 hvc_exit_stat; | |
| 154 | + u64 wfe_exit_stat; | |
| 155 | + u64 wfi_exit_stat; | |
| 156 | + u64 mmio_exit_user; | |
| 157 | + u64 mmio_exit_kernel; | |
| 158 | + u64 exits; | |
| 153 | 159 | }; |
| 154 | 160 | |
| 155 | 161 | int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init); |
| @@ -279,6 +279,11 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd, · 278 unmodified lines | ||
| 279 | 279 | pgd_t *merged_hyp_pgd, |
| 280 | 280 | unsigned long hyp_idmap_start) { } |
| 281 | 281 | |
| 282 | +static inline unsigned int kvm_get_vmid_bits(void) | |
| 283 | +{ | |
| 284 | + return 8; | |
| 285 | +} | |
| 286 | + | |
| 282 | 287 | #endif /* !__ASSEMBLY__ */ |
| 283 | 288 | |
| 284 | 289 | #endif /* __ARM_KVM_MMU_H__ */ |
| @@ -416,6 +416,7 @@ · 415 unmodified lines | ||
| 416 | 416 | #define __NR_execveat (__NR_SYSCALL_BASE+387) |
| 417 | 417 | #define __NR_userfaultfd (__NR_SYSCALL_BASE+388) |
| 418 | 418 | #define __NR_membarrier (__NR_SYSCALL_BASE+389) |
| 419 | +#define __NR_mlock2 (__NR_SYSCALL_BASE+390) | |
| 419 | 420 | |
| 420 | 421 | /* |
| 421 | 422 | * The following SWIs are ARM private. |
| @@ -17,11 +17,6 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <asm/mach/pci.h> |
| 18 | 18 | |
| 19 | 19 | static int debug_pci; |
| 20 | −static resource_size_t (*align_resource)(struct pci_dev *dev, | |
| 21 | − const struct resource *res, | |
| 22 | − resource_size_t start, | |
| 23 | − resource_size_t size, | |
| 24 | − resource_size_t align) = NULL; | |
| 25 | 20 | |
| 26 | 21 | /* |
| 27 | 22 | * We can't use pci_get_device() here since we are |
| @@ -461,7 +456,6 @@ static void pcibios_init_hw(struct device *parent, struct hw_pci *hw, · 433 unmodified lines | ||
| 461 | 456 | sys->busnr = busnr; |
| 462 | 457 | sys->swizzle = hw->swizzle; |
| 463 | 458 | sys->map_irq = hw->map_irq; |
| 464 | − align_resource = hw->align_resource; | |
| 465 | 459 | INIT_LIST_HEAD(&sys->resources); |
| 466 | 460 | |
| 467 | 461 | if (hw->private_data) |
| @@ -470,6 +464,8 @@ static void pcibios_init_hw(struct device *parent, struct hw_pci *hw, · 2 unmodified lines | ||
| 470 | 464 | ret = hw->setup(nr, sys); |
| 471 | 465 | |
| 472 | 466 | if (ret > 0) { |
| 467 | + struct pci_host_bridge *host_bridge; | |
| 468 | + | |
| 473 | 469 | ret = pcibios_init_resources(nr, sys); |
| 474 | 470 | if (ret) { |
| 475 | 471 | kfree(sys); |
| @@ -491,6 +487,9 @@ static void pcibios_init_hw(struct device *parent, struct hw_pci *hw, · 15 unmodified lines | ||
| 491 | 487 | busnr = sys->bus->busn_res.end + 1; |
| 492 | 488 | |
| 493 | 489 | list_add(&sys->node, head); |
| 490 | + | |
| 491 | + host_bridge = pci_find_host_bridge(sys->bus); | |
| 492 | + host_bridge->align_resource = hw->align_resource; | |
| 494 | 493 | } else { |
| 495 | 494 | kfree(sys); |
| 496 | 495 | if (ret < 0) |
| @@ -578,14 +577,18 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res, · 81 unmodified lines | ||
| 578 | 577 | { |
| 579 | 578 | struct pci_dev *dev = data; |
| 580 | 579 | resource_size_t start = res->start; |
| 580 | + struct pci_host_bridge *host_bridge; | |
| 581 | 581 | |
| 582 | 582 | if (res->flags & IORESOURCE_IO && start & 0x300) |
| 583 | 583 | start = (start + 0x3ff) & ~0x3ff; |
| 584 | 584 | |
| 585 | 585 | start = (start + align - 1) & ~(align - 1); |
| 586 | 586 | |
| 587 | − if (align_resource) | |
| 588 | − return align_resource(dev, res, start, size, align); | |
| 587 | + host_bridge = pci_find_host_bridge(dev->bus); | |
| 588 | + | |
| 589 | + if (host_bridge->align_resource) | |
| 590 | + return host_bridge->align_resource(dev, res, | |
| 591 | + start, size, align); | |
| 589 | 592 | |
| 590 | 593 | return start; |
| 591 | 594 | } |
| @@ -399,6 +399,7 @@ · 398 unmodified lines | ||
| 399 | 399 | CALL(sys_execveat) |
| 400 | 400 | CALL(sys_userfaultfd) |
| 401 | 401 | CALL(sys_membarrier) |
| 402 | + CALL(sys_mlock2) | |
| 402 | 403 | #ifndef syscalls_counted |
| 403 | 404 | .equ syscalls_padding, ((NR_syscalls + 3) & ~3) - NR_syscalls |
| 404 | 405 | #define syscalls_counted |
| @@ -44,6 +44,7 @@ · 43 unmodified lines | ||
| 44 | 44 | #include <asm/kvm_emulate.h> |
| 45 | 45 | #include <asm/kvm_coproc.h> |
| 46 | 46 | #include <asm/kvm_psci.h> |
| 47 | +#include <asm/sections.h> | |
| 47 | 48 | |
| 48 | 49 | #ifdef REQUIRES_VIRT |
| 49 | 50 | __asm__(".arch_extension virt"); |
| @@ -58,9 +59,12 @@ static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_arm_running_vcpu); · 8 unmodified lines | ||
| 58 | 59 | |
| 59 | 60 | /* The VMID used in the VTTBR */ |
| 60 | 61 | static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1); |
| 61 | −static u8 kvm_next_vmid; | |
| 62 | +static u32 kvm_next_vmid; | |
| 63 | +static unsigned int kvm_vmid_bits __read_mostly; | |
| 62 | 64 | static DEFINE_SPINLOCK(kvm_vmid_lock); |
| 63 | 65 | |
| 66 | +static bool vgic_present; | |
| 67 | + | |
| 64 | 68 | static void kvm_arm_set_running_vcpu(struct kvm_vcpu *vcpu) |
| 65 | 69 | { |
| 66 | 70 | BUG_ON(preemptible()); |
| @@ -132,7 +136,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · 65 unmodified lines | ||
| 132 | 136 | kvm->arch.vmid_gen = 0; |
| 133 | 137 | |
| 134 | 138 | /* The maximum number of VCPUs is limited by the host's GIC model */ |
| 135 | − kvm->arch.max_vcpus = kvm_vgic_get_max_vcpus(); | |
| 139 | + kvm->arch.max_vcpus = vgic_present ? | |
| 140 | + kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; | |
| 136 | 141 | |
| 137 | 142 | return ret; |
| 138 | 143 | out_free_stage2_pgd: |
| @@ -172,6 +177,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · 33 unmodified lines | ||
| 172 | 177 | int r; |
| 173 | 178 | switch (ext) { |
| 174 | 179 | case KVM_CAP_IRQCHIP: |
| 180 | + r = vgic_present; | |
| 181 | + break; | |
| 175 | 182 | case KVM_CAP_IOEVENTFD: |
| 176 | 183 | case KVM_CAP_DEVICE_CTRL: |
| 177 | 184 | case KVM_CAP_USER_MEMORY: |
| @@ -433,11 +440,12 @@ static void update_vttbr(struct kvm *kvm) · 255 unmodified lines | ||
| 433 | 440 | kvm->arch.vmid_gen = atomic64_read(&kvm_vmid_gen); |
| 434 | 441 | kvm->arch.vmid = kvm_next_vmid; |
| 435 | 442 | kvm_next_vmid++; |
| 443 | + kvm_next_vmid &= (1 << kvm_vmid_bits) - 1; | |
| 436 | 444 | |
| 437 | 445 | /* update vttbr to be used with the new vmid */ |
| 438 | 446 | pgd_phys = virt_to_phys(kvm_get_hwpgd(kvm)); |
| 439 | 447 | BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK); |
| 440 | − vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK; | |
| 448 | + vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits); | |
| 441 | 449 | kvm->arch.vttbr = pgd_phys | vmid; |
| 442 | 450 | |
| 443 | 451 | spin_unlock(&kvm_vmid_lock); |
| @@ -603,6 +611,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) · 159 unmodified lines | ||
| 603 | 611 | ret = kvm_call_hyp(__kvm_vcpu_run, vcpu); |
| 604 | 612 | |
| 605 | 613 | vcpu->mode = OUTSIDE_GUEST_MODE; |
| 614 | + vcpu->stat.exits++; | |
| 606 | 615 | /* |
| 607 | 616 | * Back from guest |
| 608 | 617 | *************************************************************/ |
| @@ -913,6 +922,8 @@ static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, · 304 unmodified lines | ||
| 913 | 922 | |
| 914 | 923 | switch (dev_id) { |
| 915 | 924 | case KVM_ARM_DEVICE_VGIC_V2: |
| 925 | + if (!vgic_present) | |
| 926 | + return -ENXIO; | |
| 916 | 927 | return kvm_vgic_addr(kvm, type, &dev_addr->addr, true); |
| 917 | 928 | default: |
| 918 | 929 | return -ENODEV; |
| @@ -927,6 +938,8 @@ long kvm_arch_vm_ioctl(struct file *filp, · 8 unmodified lines | ||
| 927 | 938 | |
| 928 | 939 | switch (ioctl) { |
| 929 | 940 | case KVM_CREATE_IRQCHIP: { |
| 941 | + if (!vgic_present) | |
| 942 | + return -ENXIO; | |
| 930 | 943 | return kvm_vgic_create(kvm, KVM_DEV_TYPE_ARM_VGIC_V2); |
| 931 | 944 | } |
| 932 | 945 | case KVM_ARM_SET_DEVICE_ADDR: { |
| @@ -1067,6 +1080,12 @@ static int init_hyp_mode(void) · 134 unmodified lines | ||
| 1067 | 1080 | goto out_free_mappings; |
| 1068 | 1081 | } |
| 1069 | 1082 | |
| 1083 | + err = create_hyp_mappings(__start_rodata, __end_rodata); | |
| 1084 | + if (err) { | |
| 1085 | + kvm_err("Cannot map rodata section\n"); | |
| 1086 | + goto out_free_mappings; | |
| 1087 | + } | |
| 1088 | + | |
| 1070 | 1089 | /* |
| 1071 | 1090 | * Map the Hyp stack pages |
| 1072 | 1091 | */ |
| @@ -1111,8 +1130,17 @@ static int init_hyp_mode(void) · 38 unmodified lines | ||
| 1111 | 1130 | * Init HYP view of VGIC |
| 1112 | 1131 | */ |
| 1113 | 1132 | err = kvm_vgic_hyp_init(); |
| 1114 | − if (err) | |
| 1133 | + switch (err) { | |
| 1134 | + case 0: | |
| 1135 | + vgic_present = true; | |
| 1136 | + break; | |
| 1137 | + case -ENODEV: | |
| 1138 | + case -ENXIO: | |
| 1139 | + vgic_present = false; | |
| 1140 | + break; | |
| 1141 | + default: | |
| 1115 | 1142 | goto out_free_context; |
| 1143 | + } | |
| 1116 | 1144 | |
| 1117 | 1145 | /* |
| 1118 | 1146 | * Init HYP architected timer support |
| @@ -1127,6 +1155,10 @@ static int init_hyp_mode(void) · 8 unmodified lines | ||
| 1127 | 1155 | |
| 1128 | 1156 | kvm_perf_init(); |
| 1129 | 1157 | |
| 1158 | + /* set size of VMID supported by CPU */ | |
| 1159 | + kvm_vmid_bits = kvm_get_vmid_bits(); | |
| 1160 | + kvm_info("%d-bit VMID\n", kvm_vmid_bits); | |
| 1161 | + | |
| 1130 | 1162 | kvm_info("Hyp mode initialized successfully\n"); |
| 1131 | 1163 | |
| 1132 | 1164 | return 0; |
| @@ -275,6 +275,40 @@ static u32 exc_vector_base(struct kvm_vcpu *vcpu) · 274 unmodified lines | ||
| 275 | 275 | return vbar; |
| 276 | 276 | } |
| 277 | 277 | |
| 278 | +/* | |
| 279 | + * Switch to an exception mode, updating both CPSR and SPSR. Follow | |
| 280 | + * the logic described in AArch32.EnterMode() from the ARMv8 ARM. | |
| 281 | + */ | |
| 282 | +static void kvm_update_psr(struct kvm_vcpu *vcpu, unsigned long mode) | |
| 283 | +{ | |
| 284 | + unsigned long cpsr = *vcpu_cpsr(vcpu); | |
| 285 | + u32 sctlr = vcpu->arch.cp15[c1_SCTLR]; | |
| 286 | + | |
| 287 | + *vcpu_cpsr(vcpu) = (cpsr & ~MODE_MASK) | mode; | |
| 288 | + | |
| 289 | + switch (mode) { | |
| 290 | + case FIQ_MODE: | |
| 291 | + *vcpu_cpsr(vcpu) |= PSR_F_BIT; | |
| 292 | + /* Fall through */ | |
| 293 | + case ABT_MODE: | |
| 294 | + case IRQ_MODE: | |
| 295 | + *vcpu_cpsr(vcpu) |= PSR_A_BIT; | |
| 296 | + /* Fall through */ | |
| 297 | + default: | |
| 298 | + *vcpu_cpsr(vcpu) |= PSR_I_BIT; | |
| 299 | + } | |
| 300 | + | |
| 301 | + *vcpu_cpsr(vcpu) &= ~(PSR_IT_MASK | PSR_J_BIT | PSR_E_BIT | PSR_T_BIT); | |
| 302 | + | |
| 303 | + if (sctlr & SCTLR_TE) | |
| 304 | + *vcpu_cpsr(vcpu) |= PSR_T_BIT; | |
| 305 | + if (sctlr & SCTLR_EE) | |
| 306 | + *vcpu_cpsr(vcpu) |= PSR_E_BIT; | |
| 307 | + | |
| 308 | + /* Note: These now point to the mode banked copies */ | |
| 309 | + *vcpu_spsr(vcpu) = cpsr; | |
| 310 | +} | |
| 311 | + | |
| 278 | 312 | /** |
| 279 | 313 | * kvm_inject_undefined - inject an undefined exception into the guest |
| 280 | 314 | * @vcpu: The VCPU to receive the undefined exception |
| @@ -286,29 +320,13 @@ static u32 exc_vector_base(struct kvm_vcpu *vcpu) · 5 unmodified lines | ||
| 286 | 320 | */ |
| 287 | 321 | void kvm_inject_undefined(struct kvm_vcpu *vcpu) |
| 288 | 322 | { |
| 289 | − unsigned long new_lr_value; | |
| 290 | − unsigned long new_spsr_value; | |
| 291 | 323 | unsigned long cpsr = *vcpu_cpsr(vcpu); |
| 292 | − u32 sctlr = vcpu->arch.cp15[c1_SCTLR]; | |
| 293 | 324 | bool is_thumb = (cpsr & PSR_T_BIT); |
| 294 | 325 | u32 vect_offset = 4; |
| 295 | 326 | u32 return_offset = (is_thumb) ? 2 : 4; |
| 296 | 327 | |
| 297 | − new_spsr_value = cpsr; | |
| 298 | − new_lr_value = *vcpu_pc(vcpu) - return_offset; | |
| 299 | − | |
| 300 | − *vcpu_cpsr(vcpu) = (cpsr & ~MODE_MASK) | UND_MODE; | |
| 301 | − *vcpu_cpsr(vcpu) |= PSR_I_BIT; | |
| 302 | − *vcpu_cpsr(vcpu) &= ~(PSR_IT_MASK | PSR_J_BIT | PSR_E_BIT | PSR_T_BIT); | |
| 303 | − | |
| 304 | − if (sctlr & SCTLR_TE) | |
| 305 | − *vcpu_cpsr(vcpu) |= PSR_T_BIT; | |
| 306 | − if (sctlr & SCTLR_EE) | |
| 307 | − *vcpu_cpsr(vcpu) |= PSR_E_BIT; | |
| 308 | − | |
| 309 | − /* Note: These now point to UND banked copies */ | |
| 310 | − *vcpu_spsr(vcpu) = cpsr; | |
| 311 | − *vcpu_reg(vcpu, 14) = new_lr_value; | |
| 328 | + kvm_update_psr(vcpu, UND_MODE); | |
| 329 | + *vcpu_reg(vcpu, 14) = *vcpu_pc(vcpu) - return_offset; | |
| 312 | 330 | |
| 313 | 331 | /* Branch to exception vector */ |
| 314 | 332 | *vcpu_pc(vcpu) = exc_vector_base(vcpu) + vect_offset; |
| @@ -320,30 +338,14 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu) · 5 unmodified lines | ||
| 320 | 338 | */ |
| 321 | 339 | static void inject_abt(struct kvm_vcpu *vcpu, bool is_pabt, unsigned long addr) |
| 322 | 340 | { |
| 323 | − unsigned long new_lr_value; | |
| 324 | − unsigned long new_spsr_value; | |
| 325 | 341 | unsigned long cpsr = *vcpu_cpsr(vcpu); |
| 326 | − u32 sctlr = vcpu->arch.cp15[c1_SCTLR]; | |
| 327 | 342 | bool is_thumb = (cpsr & PSR_T_BIT); |
| 328 | 343 | u32 vect_offset; |
| 329 | 344 | u32 return_offset = (is_thumb) ? 4 : 0; |
| 330 | 345 | bool is_lpae; |
| 331 | 346 | |
| 332 | − new_spsr_value = cpsr; | |
| 333 | − new_lr_value = *vcpu_pc(vcpu) + return_offset; | |
| 334 | − | |
| 335 | − *vcpu_cpsr(vcpu) = (cpsr & ~MODE_MASK) | ABT_MODE; | |
| 336 | − *vcpu_cpsr(vcpu) |= PSR_I_BIT | PSR_A_BIT; | |
| 337 | − *vcpu_cpsr(vcpu) &= ~(PSR_IT_MASK | PSR_J_BIT | PSR_E_BIT | PSR_T_BIT); | |
| 338 | − | |
| 339 | − if (sctlr & SCTLR_TE) | |
| 340 | − *vcpu_cpsr(vcpu) |= PSR_T_BIT; | |
| 341 | − if (sctlr & SCTLR_EE) | |
| 342 | − *vcpu_cpsr(vcpu) |= PSR_E_BIT; | |
| 343 | − | |
| 344 | − /* Note: These now point to ABT banked copies */ | |
| 345 | − *vcpu_spsr(vcpu) = cpsr; | |
| 346 | − *vcpu_reg(vcpu, 14) = new_lr_value; | |
| 347 | + kvm_update_psr(vcpu, ABT_MODE); | |
| 348 | + *vcpu_reg(vcpu, 14) = *vcpu_pc(vcpu) + return_offset; | |
| 347 | 349 | |
| 348 | 350 | if (is_pabt) |
| 349 | 351 | vect_offset = 12; |
| @@ -33,6 +33,12 @@ · 32 unmodified lines | ||
| 33 | 33 | #define VCPU_STAT(x) { #x, offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU } |
| 34 | 34 | |
| 35 | 35 | struct kvm_stats_debugfs_item debugfs_entries[] = { |
| 36 | + VCPU_STAT(hvc_exit_stat), | |
| 37 | + VCPU_STAT(wfe_exit_stat), | |
| 38 | + VCPU_STAT(wfi_exit_stat), | |
| 39 | + VCPU_STAT(mmio_exit_user), | |
| 40 | + VCPU_STAT(mmio_exit_kernel), | |
| 41 | + VCPU_STAT(exits), | |
| 36 | 42 | { NULL } |
| 37 | 43 | }; |
| 38 | 44 | |
| @@ -42,6 +42,7 @@ static int handle_hvc(struct kvm_vcpu *vcpu, struct kvm_run *run) · 41 unmodified lines | ||
| 42 | 42 | |
| 43 | 43 | trace_kvm_hvc(*vcpu_pc(vcpu), *vcpu_reg(vcpu, 0), |
| 44 | 44 | kvm_vcpu_hvc_get_imm(vcpu)); |
| 45 | + vcpu->stat.hvc_exit_stat++; | |
| 45 | 46 | |
| 46 | 47 | ret = kvm_psci_call(vcpu); |
| 47 | 48 | if (ret < 0) { |
| @@ -89,9 +90,11 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run) · 41 unmodified lines | ||
| 89 | 90 | { |
| 90 | 91 | if (kvm_vcpu_get_hsr(vcpu) & HSR_WFI_IS_WFE) { |
| 91 | 92 | trace_kvm_wfx(*vcpu_pc(vcpu), true); |
| 93 | + vcpu->stat.wfe_exit_stat++; | |
| 92 | 94 | kvm_vcpu_on_spin(vcpu); |
| 93 | 95 | } else { |
| 94 | 96 | trace_kvm_wfx(*vcpu_pc(vcpu), false); |
| 97 | + vcpu->stat.wfi_exit_stat++; | |
| 95 | 98 | kvm_vcpu_block(vcpu); |
| 96 | 99 | } |
| 97 | 100 | |
| @@ -115,7 +115,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu, struct kvm_run *run) · 114 unmodified lines | ||
| 115 | 115 | trace_kvm_mmio(KVM_TRACE_MMIO_READ, len, run->mmio.phys_addr, |
| 116 | 116 | data); |
| 117 | 117 | data = vcpu_data_host_to_guest(vcpu, data, len); |
| 118 | − *vcpu_reg(vcpu, vcpu->arch.mmio_decode.rt) = data; | |
| 118 | + vcpu_set_reg(vcpu, vcpu->arch.mmio_decode.rt, data); | |
| 119 | 119 | } |
| 120 | 120 | |
| 121 | 121 | return 0; |
| @@ -186,7 +186,8 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, · 64 unmodified lines | ||
| 186 | 186 | rt = vcpu->arch.mmio_decode.rt; |
| 187 | 187 | |
| 188 | 188 | if (is_write) { |
| 189 | − data = vcpu_data_guest_to_host(vcpu, *vcpu_reg(vcpu, rt), len); | |
| 189 | + data = vcpu_data_guest_to_host(vcpu, vcpu_get_reg(vcpu, rt), | |
| 190 | + len); | |
| 190 | 191 | |
| 191 | 192 | trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, fault_ipa, data); |
| 192 | 193 | mmio_write_buf(data_buf, len, data); |
| @@ -209,8 +210,11 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, · 16 unmodified lines | ||
| 209 | 210 | |
| 210 | 211 | if (!ret) { |
| 211 | 212 | /* We handled the access successfully in the kernel. */ |
| 213 | + vcpu->stat.mmio_exit_kernel++; | |
| 212 | 214 | kvm_handle_mmio_return(vcpu, run); |
| 213 | 215 | return 1; |
| 216 | + } else { | |
| 217 | + vcpu->stat.mmio_exit_user++; | |
| 214 | 218 | } |
| 215 | 219 | |
| 216 | 220 | run->exit_reason = KVM_EXIT_MMIO; |
| @@ -218,7 +218,7 @@ static void unmap_ptes(struct kvm *kvm, pmd_t *pmd, · 217 unmodified lines | ||
| 218 | 218 | kvm_tlb_flush_vmid_ipa(kvm, addr); |
| 219 | 219 | |
| 220 | 220 | /* No need to invalidate the cache for device mappings */ |
| 221 | − if (!kvm_is_device_pfn(__phys_to_pfn(addr))) | |
| 221 | + if (!kvm_is_device_pfn(pte_pfn(old_pte))) | |
| 222 | 222 | kvm_flush_dcache_pte(old_pte); |
| 223 | 223 | |
| 224 | 224 | put_page(virt_to_page(pte)); |
| @@ -310,7 +310,7 @@ static void stage2_flush_ptes(struct kvm *kvm, pmd_t *pmd, · 85 unmodified lines | ||
| 310 | 310 | |
| 311 | 311 | pte = pte_offset_kernel(pmd, addr); |
| 312 | 312 | do { |
| 313 | − if (!pte_none(*pte) && !kvm_is_device_pfn(__phys_to_pfn(addr))) | |
| 313 | + if (!pte_none(*pte) && !kvm_is_device_pfn(pte_pfn(*pte))) | |
| 314 | 314 | kvm_flush_dcache_pte(*pte); |
| 315 | 315 | } while (pte++, addr += PAGE_SIZE, addr != end); |
| 316 | 316 | } |
| @@ -656,9 +656,9 @@ static void *kvm_alloc_hwpgd(void) · 339 unmodified lines | ||
| 656 | 656 | * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation. |
| 657 | 657 | * @kvm: The KVM struct pointer for the VM. |
| 658 | 658 | * |
| 659 | − * Allocates the 1st level table only of size defined by S2_PGD_ORDER (can | |
| 660 | − * support either full 40-bit input addresses or limited to 32-bit input | |
| 661 | − * addresses). Clears the allocated pages. | |
| 659 | + * Allocates only the stage-2 HW PGD level table(s) (can support either full | |
| 660 | + * 40-bit input addresses or limited to 32-bit input addresses). Clears the | |
| 661 | + * allocated pages. | |
| 662 | 662 | * |
| 663 | 663 | * Note we don't need locking here as this is only called when the VM is |
| 664 | 664 | * created, which can only be done once. |
| @@ -75,7 +75,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) · 74 unmodified lines | ||
| 75 | 75 | unsigned long context_id; |
| 76 | 76 | phys_addr_t target_pc; |
| 77 | 77 | |
| 78 | − cpu_id = *vcpu_reg(source_vcpu, 1) & MPIDR_HWID_BITMASK; | |
| 78 | + cpu_id = vcpu_get_reg(source_vcpu, 1) & MPIDR_HWID_BITMASK; | |
| 79 | 79 | if (vcpu_mode_is_32bit(source_vcpu)) |
| 80 | 80 | cpu_id &= ~((u32) 0); |
| 81 | 81 | |
| @@ -94,8 +94,8 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) · 12 unmodified lines | ||
| 94 | 94 | return PSCI_RET_INVALID_PARAMS; |
| 95 | 95 | } |
| 96 | 96 | |
| 97 | − target_pc = *vcpu_reg(source_vcpu, 2); | |
| 98 | − context_id = *vcpu_reg(source_vcpu, 3); | |
| 97 | + target_pc = vcpu_get_reg(source_vcpu, 2); | |
| 98 | + context_id = vcpu_get_reg(source_vcpu, 3); | |
| 99 | 99 | |
| 100 | 100 | kvm_reset_vcpu(vcpu); |
| 101 | 101 | |
| @@ -114,7 +114,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) · 12 unmodified lines | ||
| 114 | 114 | * NOTE: We always update r0 (or x0) because for PSCI v0.1 |
| 115 | 115 | * the general puspose registers are undefined upon CPU_ON. |
| 116 | 116 | */ |
| 117 | − *vcpu_reg(vcpu, 0) = context_id; | |
| 117 | + vcpu_set_reg(vcpu, 0, context_id); | |
| 118 | 118 | vcpu->arch.power_off = false; |
| 119 | 119 | smp_mb(); /* Make sure the above is visible */ |
| 120 | 120 | |
| @@ -134,8 +134,8 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu) · 13 unmodified lines | ||
| 134 | 134 | struct kvm *kvm = vcpu->kvm; |
| 135 | 135 | struct kvm_vcpu *tmp; |
| 136 | 136 | |
| 137 | − target_affinity = *vcpu_reg(vcpu, 1); | |
| 138 | − lowest_affinity_level = *vcpu_reg(vcpu, 2); | |
| 137 | + target_affinity = vcpu_get_reg(vcpu, 1); | |
| 138 | + lowest_affinity_level = vcpu_get_reg(vcpu, 2); | |
| 139 | 139 | |
| 140 | 140 | /* Determine target affinity mask */ |
| 141 | 141 | target_affinity_mask = psci_affinity_mask(lowest_affinity_level); |
| @@ -209,7 +209,7 @@ int kvm_psci_version(struct kvm_vcpu *vcpu) · 67 unmodified lines | ||
| 209 | 209 | static int kvm_psci_0_2_call(struct kvm_vcpu *vcpu) |
| 210 | 210 | { |
| 211 | 211 | int ret = 1; |
| 212 | − unsigned long psci_fn = *vcpu_reg(vcpu, 0) & ~((u32) 0); | |
| 212 | + unsigned long psci_fn = vcpu_get_reg(vcpu, 0) & ~((u32) 0); | |
| 213 | 213 | unsigned long val; |
| 214 | 214 | |
| 215 | 215 | switch (psci_fn) { |
| @@ -273,13 +273,13 @@ static int kvm_psci_0_2_call(struct kvm_vcpu *vcpu) · 57 unmodified lines | ||
| 273 | 273 | break; |
| 274 | 274 | } |
| 275 | 275 | |
| 276 | − *vcpu_reg(vcpu, 0) = val; | |
| 276 | + vcpu_set_reg(vcpu, 0, val); | |
| 277 | 277 | return ret; |
| 278 | 278 | } |
| 279 | 279 | |
| 280 | 280 | static int kvm_psci_0_1_call(struct kvm_vcpu *vcpu) |
| 281 | 281 | { |
| 282 | − unsigned long psci_fn = *vcpu_reg(vcpu, 0) & ~((u32) 0); | |
| 282 | + unsigned long psci_fn = vcpu_get_reg(vcpu, 0) & ~((u32) 0); | |
| 283 | 283 | unsigned long val; |
| 284 | 284 | |
| 285 | 285 | switch (psci_fn) { |
| @@ -295,7 +295,7 @@ static int kvm_psci_0_1_call(struct kvm_vcpu *vcpu) · 9 unmodified lines | ||
| 295 | 295 | break; |
| 296 | 296 | } |
| 297 | 297 | |
| 298 | − *vcpu_reg(vcpu, 0) = val; | |
| 298 | + vcpu_set_reg(vcpu, 0, val); | |
| 299 | 299 | return 1; |
| 300 | 300 | } |
| 301 | 301 | |
| @@ -4,7 +4,6 @@ menuconfig ARCH_AT91 · 3 unmodified lines | ||
| 4 | 4 | select ARCH_REQUIRE_GPIOLIB |
| 5 | 5 | select COMMON_CLK_AT91 |
| 6 | 6 | select PINCTRL |
| 7 | − select PINCTRL_AT91 | |
| 8 | 7 | select SOC_BUS |
| 9 | 8 | |
| 10 | 9 | if ARCH_AT91 |
| @@ -17,6 +16,7 @@ config SOC_SAMA5D2 · 6 unmodified lines | ||
| 17 | 16 | select HAVE_AT91_USB_CLK |
| 18 | 17 | select HAVE_AT91_H32MX |
| 19 | 18 | select HAVE_AT91_GENERATED_CLK |
| 19 | + select PINCTRL_AT91PIO4 | |
| 20 | 20 | help |
| 21 | 21 | Select this if ou are using one of Atmel's SAMA5D2 family SoC. |
| 22 | 22 | |
| @@ -27,6 +27,7 @@ config SOC_SAMA5D3 · 4 unmodified lines | ||
| 27 | 27 | select HAVE_AT91_UTMI |
| 28 | 28 | select HAVE_AT91_SMD |
| 29 | 29 | select HAVE_AT91_USB_CLK |
| 30 | + select PINCTRL_AT91 | |
| 30 | 31 | help |
| 31 | 32 | Select this if you are using one of Atmel's SAMA5D3 family SoC. |
| 32 | 33 | This support covers SAMA5D31, SAMA5D33, SAMA5D34, SAMA5D35, SAMA5D36. |
| @@ -40,6 +41,7 @@ config SOC_SAMA5D4 · 7 unmodified lines | ||
| 40 | 41 | select HAVE_AT91_SMD |
| 41 | 42 | select HAVE_AT91_USB_CLK |
| 42 | 43 | select HAVE_AT91_H32MX |
| 44 | + select PINCTRL_AT91 | |
| 43 | 45 | help |
| 44 | 46 | Select this if you are using one of Atmel's SAMA5D4 family SoC. |
| 45 | 47 | |
| @@ -50,6 +52,7 @@ config SOC_AT91RM9200 · 4 unmodified lines | ||
| 50 | 52 | select CPU_ARM920T |
| 51 | 53 | select HAVE_AT91_USB_CLK |
| 52 | 54 | select MIGHT_HAVE_PCI |
| 55 | + select PINCTRL_AT91 | |
| 53 | 56 | select SOC_SAM_V4_V5 |
| 54 | 57 | select SRAM if PM |
| 55 | 58 | help |
| @@ -65,6 +68,7 @@ config SOC_AT91SAM9 · 9 unmodified lines | ||
| 65 | 68 | select HAVE_AT91_UTMI |
| 66 | 69 | select HAVE_FB_ATMEL |
| 67 | 70 | select MEMORY |
| 71 | + select PINCTRL_AT91 | |
| 68 | 72 | select SOC_SAM_V4_V5 |
| 69 | 73 | select SRAM if PM |
| 70 | 74 | help |
| @@ -41,8 +41,10 @@ · 40 unmodified lines | ||
| 41 | 41 | * implementation should be moved down into the pinctrl driver and get |
| 42 | 42 | * called as part of the generic suspend/resume path. |
| 43 | 43 | */ |
| 44 | +#ifdef CONFIG_PINCTRL_AT91 | |
| 44 | 45 | extern void at91_pinctrl_gpio_suspend(void); |
| 45 | 46 | extern void at91_pinctrl_gpio_resume(void); |
| 47 | +#endif | |
| 46 | 48 | |
| 47 | 49 | static struct { |
| 48 | 50 | unsigned long uhp_udp_mask; |
| @@ -151,8 +153,9 @@ static void at91_pm_suspend(suspend_state_t state) · 102 unmodified lines | ||
| 151 | 153 | |
| 152 | 154 | static int at91_pm_enter(suspend_state_t state) |
| 153 | 155 | { |
| 156 | +#ifdef CONFIG_PINCTRL_AT91 | |
| 154 | 157 | at91_pinctrl_gpio_suspend(); |
| 155 | − | |
| 158 | +#endif | |
| 156 | 159 | switch (state) { |
| 157 | 160 | /* |
| 158 | 161 | * Suspend-to-RAM is like STANDBY plus slow clock mode, so |
| @@ -192,7 +195,9 @@ static int at91_pm_enter(suspend_state_t state) · 33 unmodified lines | ||
| 192 | 195 | error: |
| 193 | 196 | target_state = PM_SUSPEND_ON; |
| 194 | 197 | |
| 198 | +#ifdef CONFIG_PINCTRL_AT91 | |
| 195 | 199 | at91_pinctrl_gpio_resume(); |
| 200 | +#endif | |
| 196 | 201 | return 0; |
| 197 | 202 | } |
| 198 | 203 | |
| @@ -18,13 +18,13 @@ · 17 unmodified lines | ||
| 18 | 18 | @ check low interrupts |
| 19 | 19 | ldr \irqstat, [\base, #IRQ_CAUSE_LOW_OFF] |
| 20 | 20 | ldr \tmp, [\base, #IRQ_MASK_LOW_OFF] |
| 21 | − mov \irqnr, #31 | |
| 21 | + mov \irqnr, #32 | |
| 22 | 22 | ands \irqstat, \irqstat, \tmp |
| 23 | 23 | |
| 24 | 24 | @ if no low interrupts set, check high interrupts |
| 25 | 25 | ldreq \irqstat, [\base, #IRQ_CAUSE_HIGH_OFF] |
| 26 | 26 | ldreq \tmp, [\base, #IRQ_MASK_HIGH_OFF] |
| 27 | − moveq \irqnr, #63 | |
| 27 | + moveq \irqnr, #64 | |
| 28 | 28 | andeqs \irqstat, \irqstat, \tmp |
| 29 | 29 | |
| 30 | 30 | @ find first active interrupt source |
| @@ -748,8 +748,12 @@ static void exynos5_powerdown_conf(enum sys_powerdown mode) · 747 unmodified lines | ||
| 748 | 748 | void exynos_sys_powerdown_conf(enum sys_powerdown mode) |
| 749 | 749 | { |
| 750 | 750 | unsigned int i; |
| 751 | + const struct exynos_pmu_data *pmu_data; | |
| 752 | + | |
| 753 | + if (!pmu_context) | |
| 754 | + return; | |
| 751 | 755 | |
| 752 | − const struct exynos_pmu_data *pmu_data = pmu_context->pmu_data; | |
| 756 | + pmu_data = pmu_context->pmu_data; | |
| 753 | 757 | |
| 754 | 758 | if (pmu_data->powerdown_conf) |
| 755 | 759 | pmu_data->powerdown_conf(mode); |
| @@ -177,6 +177,7 @@ static struct irq_chip imx_gpc_chip = { · 176 unmodified lines | ||
| 177 | 177 | .irq_unmask = imx_gpc_irq_unmask, |
| 178 | 178 | .irq_retrigger = irq_chip_retrigger_hierarchy, |
| 179 | 179 | .irq_set_wake = imx_gpc_irq_set_wake, |
| 180 | + .irq_set_type = irq_chip_set_type_parent, | |
| 180 | 181 | #ifdef CONFIG_SMP |
| 181 | 182 | .irq_set_affinity = irq_chip_set_affinity_parent, |
| 182 | 183 | #endif |
| @@ -143,7 +143,7 @@ static inline void __indirect_writesl(volatile void __iomem *bus_addr, · 142 unmodified lines | ||
| 143 | 143 | writel(*vaddr++, bus_addr); |
| 144 | 144 | } |
| 145 | 145 | |
| 146 | −static inline unsigned char __indirect_readb(const volatile void __iomem *p) | |
| 146 | +static inline u8 __indirect_readb(const volatile void __iomem *p) | |
| 147 | 147 | { |
| 148 | 148 | u32 addr = (u32)p; |
| 149 | 149 | u32 n, byte_enables, data; |
| @@ -166,7 +166,7 @@ static inline void __indirect_readsb(const volatile void __iomem *bus_addr, · 16 unmodified lines | ||
| 166 | 166 | *vaddr++ = readb(bus_addr); |
| 167 | 167 | } |
| 168 | 168 | |
| 169 | −static inline unsigned short __indirect_readw(const volatile void __iomem *p) | |
| 169 | +static inline u16 __indirect_readw(const volatile void __iomem *p) | |
| 170 | 170 | { |
| 171 | 171 | u32 addr = (u32)p; |
| 172 | 172 | u32 n, byte_enables, data; |
| @@ -189,7 +189,7 @@ static inline void __indirect_readsw(const volatile void __iomem *bus_addr, · 16 unmodified lines | ||
| 189 | 189 | *vaddr++ = readw(bus_addr); |
| 190 | 190 | } |
| 191 | 191 | |
| 192 | −static inline unsigned long __indirect_readl(const volatile void __iomem *p) | |
| 192 | +static inline u32 __indirect_readl(const volatile void __iomem *p) | |
| 193 | 193 | { |
| 194 | 194 | u32 addr = (__force u32)p; |
| 195 | 195 | u32 data; |
| @@ -350,7 +350,7 @@ static inline void insl(u32 io_addr, void *p, u32 count) · 154 unmodified lines | ||
| 350 | 350 | ((unsigned long)p <= (PIO_MASK + PIO_OFFSET))) |
| 351 | 351 | |
| 352 | 352 | #define ioread8(p) ioread8(p) |
| 353 | −static inline unsigned int ioread8(const void __iomem *addr) | |
| 353 | +static inline u8 ioread8(const void __iomem *addr) | |
| 354 | 354 | { |
| 355 | 355 | unsigned long port = (unsigned long __force)addr; |
| 356 | 356 | if (__is_io_address(port)) |
| @@ -378,7 +378,7 @@ static inline void ioread8_rep(const void __iomem *addr, void *vaddr, u32 count) · 21 unmodified lines | ||
| 378 | 378 | } |
| 379 | 379 | |
| 380 | 380 | #define ioread16(p) ioread16(p) |
| 381 | −static inline unsigned int ioread16(const void __iomem *addr) | |
| 381 | +static inline u16 ioread16(const void __iomem *addr) | |
| 382 | 382 | { |
| 383 | 383 | unsigned long port = (unsigned long __force)addr; |
| 384 | 384 | if (__is_io_address(port)) |
| @@ -407,7 +407,7 @@ static inline void ioread16_rep(const void __iomem *addr, void *vaddr, · 22 unmodified lines | ||
| 407 | 407 | } |
| 408 | 408 | |
| 409 | 409 | #define ioread32(p) ioread32(p) |
| 410 | −static inline unsigned int ioread32(const void __iomem *addr) | |
| 410 | +static inline u32 ioread32(const void __iomem *addr) | |
| 411 | 411 | { |
| 412 | 412 | unsigned long port = (unsigned long __force)addr; |
| 413 | 413 | if (__is_io_address(port)) |
| @@ -121,6 +121,7 @@ config ARCH_OMAP2PLUS_TYPICAL · 120 unmodified lines | ||
| 121 | 121 | select NEON if CPU_V7 |
| 122 | 122 | select PM |
| 123 | 123 | select REGULATOR |
| 124 | + select REGULATOR_FIXED_VOLTAGE | |
| 124 | 125 | select TWL4030_CORE if ARCH_OMAP3 || ARCH_OMAP4 |
| 125 | 126 | select TWL4030_POWER if ARCH_OMAP3 || ARCH_OMAP4 |
| 126 | 127 | select VFP |
| @@ -201,7 +202,6 @@ config MACH_OMAP3_PANDORA · 74 unmodified lines | ||
| 201 | 202 | depends on ARCH_OMAP3 |
| 202 | 203 | default y |
| 203 | 204 | select OMAP_PACKAGE_CBB |
| 204 | − select REGULATOR_FIXED_VOLTAGE if REGULATOR | |
| 205 | 205 | |
| 206 | 206 | config MACH_NOKIA_N810 |
| 207 | 207 | bool |
| @@ -143,9 +143,9 @@ static int omap4_boot_secondary(unsigned int cpu, struct task_struct *idle) · 142 unmodified lines | ||
| 143 | 143 | * Ensure that CPU power state is set to ON to avoid CPU |
| 144 | 144 | * powerdomain transition on wfi |
| 145 | 145 | */ |
| 146 | − clkdm_wakeup(cpu1_clkdm); | |
| 147 | − omap_set_pwrdm_state(cpu1_pwrdm, PWRDM_POWER_ON); | |
| 148 | − clkdm_allow_idle(cpu1_clkdm); | |
| 146 | + clkdm_wakeup_nolock(cpu1_clkdm); | |
| 147 | + pwrdm_set_next_pwrst(cpu1_pwrdm, PWRDM_POWER_ON); | |
| 148 | + clkdm_allow_idle_nolock(cpu1_clkdm); | |
| 149 | 149 | |
| 150 | 150 | if (IS_PM44XX_ERRATUM(PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD)) { |
| 151 | 151 | while (gic_dist_disabled()) { |
| @@ -890,6 +890,36 @@ static int _init_opt_clks(struct omap_hwmod *oh) · 889 unmodified lines | ||
| 890 | 890 | return ret; |
| 891 | 891 | } |
| 892 | 892 | |
| 893 | +static void _enable_optional_clocks(struct omap_hwmod *oh) | |
| 894 | +{ | |
| 895 | + struct omap_hwmod_opt_clk *oc; | |
| 896 | + int i; | |
| 897 | + | |
| 898 | + pr_debug("omap_hwmod: %s: enabling optional clocks\n", oh->name); | |
| 899 | + | |
| 900 | + for (i = oh->opt_clks_cnt, oc = oh->opt_clks; i > 0; i--, oc++) | |
| 901 | + if (oc->_clk) { | |
| 902 | + pr_debug("omap_hwmod: enable %s:%s\n", oc->role, | |
| 903 | + __clk_get_name(oc->_clk)); | |
| 904 | + clk_enable(oc->_clk); | |
| 905 | + } | |
| 906 | +} | |
| 907 | + | |
| 908 | +static void _disable_optional_clocks(struct omap_hwmod *oh) | |
| 909 | +{ | |
| 910 | + struct omap_hwmod_opt_clk *oc; | |
| 911 | + int i; | |
| 912 | + | |
| 913 | + pr_debug("omap_hwmod: %s: disabling optional clocks\n", oh->name); | |
| 914 | + | |
| 915 | + for (i = oh->opt_clks_cnt, oc = oh->opt_clks; i > 0; i--, oc++) | |
| 916 | + if (oc->_clk) { | |
| 917 | + pr_debug("omap_hwmod: disable %s:%s\n", oc->role, | |
| 918 | + __clk_get_name(oc->_clk)); | |
| 919 | + clk_disable(oc->_clk); | |
| 920 | + } | |
| 921 | +} | |
| 922 | + | |
| 893 | 923 | /** |
| 894 | 924 | * _enable_clocks - enable hwmod main clock and interface clocks |
| 895 | 925 | * @oh: struct omap_hwmod * |
| @@ -917,6 +947,9 @@ static int _enable_clocks(struct omap_hwmod *oh) · 21 unmodified lines | ||
| 917 | 947 | clk_enable(os->_clk); |
| 918 | 948 | } |
| 919 | 949 | |
| 950 | + if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | |
| 951 | + _enable_optional_clocks(oh); | |
| 952 | + | |
| 920 | 953 | /* The opt clocks are controlled by the device driver. */ |
| 921 | 954 | |
| 922 | 955 | return 0; |
| @@ -948,41 +981,14 @@ static int _disable_clocks(struct omap_hwmod *oh) · 25 unmodified lines | ||
| 948 | 981 | clk_disable(os->_clk); |
| 949 | 982 | } |
| 950 | 983 | |
| 984 | + if (oh->flags & HWMOD_OPT_CLKS_NEEDED) | |
| 985 | + _disable_optional_clocks(oh); | |
| 986 | + | |
| 951 | 987 | /* The opt clocks are controlled by the device driver. */ |
| 952 | 988 | |
| 953 | 989 | return 0; |
| 954 | 990 | } |
| 955 | 991 | |
| 956 | −static void _enable_optional_clocks(struct omap_hwmod *oh) | |
| 957 | −{ | |
| 958 | − struct omap_hwmod_opt_clk *oc; | |
| 959 | − int i; | |
| 960 | − | |
| 961 | − pr_debug("omap_hwmod: %s: enabling optional clocks\n", oh->name); | |
| 962 | − | |
| 963 | − for (i = oh->opt_clks_cnt, oc = oh->opt_clks; i > 0; i--, oc++) | |
| 964 | − if (oc->_clk) { | |
| 965 | − pr_debug("omap_hwmod: enable %s:%s\n", oc->role, | |
| 966 | − __clk_get_name(oc->_clk)); | |
| 967 | − clk_enable(oc->_clk); | |
| 968 | − } | |
| 969 | −} | |
| 970 | − | |
| 971 | −static void _disable_optional_clocks(struct omap_hwmod *oh) | |
| 972 | −{ | |
| 973 | − struct omap_hwmod_opt_clk *oc; | |
| 974 | − int i; | |
| 975 | − | |
| 976 | − pr_debug("omap_hwmod: %s: disabling optional clocks\n", oh->name); | |
| 977 | − | |
| 978 | − for (i = oh->opt_clks_cnt, oc = oh->opt_clks; i > 0; i--, oc++) | |
| 979 | − if (oc->_clk) { | |
| 980 | − pr_debug("omap_hwmod: disable %s:%s\n", oc->role, | |
| 981 | − __clk_get_name(oc->_clk)); | |
| 982 | − clk_disable(oc->_clk); | |
| 983 | − } | |
| 984 | −} | |
| 985 | − | |
| 986 | 992 | /** |
| 987 | 993 | * _omap4_enable_module - enable CLKCTRL modulemode on OMAP4 |
| 988 | 994 | * @oh: struct omap_hwmod * |
| @@ -523,6 +523,8 @@ struct omap_hwmod_omap4_prcm { · 522 unmodified lines | ||
| 523 | 523 | * HWMOD_RECONFIG_IO_CHAIN: omap_hwmod code needs to reconfigure wake-up |
| 524 | 524 | * events by calling _reconfigure_io_chain() when a device is enabled |
| 525 | 525 | * or idled. |
| 526 | + * HWMOD_OPT_CLKS_NEEDED: The optional clocks are needed for the module to | |
| 527 | + * operate and they need to be handled at the same time as the main_clk. | |
| 526 | 528 | */ |
| 527 | 529 | #define HWMOD_SWSUP_SIDLE (1 << 0) |
| 528 | 530 | #define HWMOD_SWSUP_MSTANDBY (1 << 1) |
| @@ -538,6 +540,7 @@ struct omap_hwmod_omap4_prcm { · 9 unmodified lines | ||
| 538 | 540 | #define HWMOD_FORCE_MSTANDBY (1 << 11) |
| 539 | 541 | #define HWMOD_SWSUP_SIDLE_ACT (1 << 12) |
| 540 | 542 | #define HWMOD_RECONFIG_IO_CHAIN (1 << 13) |
| 543 | +#define HWMOD_OPT_CLKS_NEEDED (1 << 14) | |
| 541 | 544 | |
| 542 | 545 | /* |
| 543 | 546 | * omap_hwmod._int_flags definitions |
| @@ -1297,6 +1297,44 @@ static struct omap_hwmod dra7xx_mcspi4_hwmod = { · 1296 unmodified lines | ||
| 1297 | 1297 | .dev_attr = &mcspi4_dev_attr, |
| 1298 | 1298 | }; |
| 1299 | 1299 | |
| 1300 | +/* | |
| 1301 | + * 'mcasp' class | |
| 1302 | + * | |
| 1303 | + */ | |
| 1304 | +static struct omap_hwmod_class_sysconfig dra7xx_mcasp_sysc = { | |
| 1305 | + .sysc_offs = 0x0004, | |
| 1306 | + .sysc_flags = SYSC_HAS_SIDLEMODE, | |
| 1307 | + .idlemodes = (SIDLE_FORCE | SIDLE_NO | SIDLE_SMART), | |
| 1308 | + .sysc_fields = &omap_hwmod_sysc_type3, | |
| 1309 | +}; | |
| 1310 | + | |
| 1311 | +static struct omap_hwmod_class dra7xx_mcasp_hwmod_class = { | |
| 1312 | + .name = "mcasp", | |
| 1313 | + .sysc = &dra7xx_mcasp_sysc, | |
| 1314 | +}; | |
| 1315 | + | |
| 1316 | +/* mcasp3 */ | |
| 1317 | +static struct omap_hwmod_opt_clk mcasp3_opt_clks[] = { | |
| 1318 | + { .role = "ahclkx", .clk = "mcasp3_ahclkx_mux" }, | |
| 1319 | +}; | |
| 1320 | + | |
| 1321 | +static struct omap_hwmod dra7xx_mcasp3_hwmod = { | |
| 1322 | + .name = "mcasp3", | |
| 1323 | + .class = &dra7xx_mcasp_hwmod_class, | |
| 1324 | + .clkdm_name = "l4per2_clkdm", | |
| 1325 | + .main_clk = "mcasp3_aux_gfclk_mux", | |
| 1326 | + .flags = HWMOD_OPT_CLKS_NEEDED, | |
| 1327 | + .prcm = { | |
| 1328 | + .omap4 = { | |
| 1329 | + .clkctrl_offs = DRA7XX_CM_L4PER2_MCASP3_CLKCTRL_OFFSET, | |
| 1330 | + .context_offs = DRA7XX_RM_L4PER2_MCASP3_CONTEXT_OFFSET, | |
| 1331 | + .modulemode = MODULEMODE_SWCTRL, | |
| 1332 | + }, | |
| 1333 | + }, | |
| 1334 | + .opt_clks = mcasp3_opt_clks, | |
| 1335 | + .opt_clks_cnt = ARRAY_SIZE(mcasp3_opt_clks), | |
| 1336 | +}; | |
| 1337 | + | |
| 1300 | 1338 | /* |
| 1301 | 1339 | * 'mmc' class |
| 1302 | 1340 | * |
| @@ -2566,6 +2604,22 @@ static struct omap_hwmod_ocp_if dra7xx_l3_main_1__hdmi = { · 1263 unmodified lines | ||
| 2566 | 2604 | .user = OCP_USER_MPU | OCP_USER_SDMA, |
| 2567 | 2605 | }; |
| 2568 | 2606 | |
| 2607 | +/* l4_per2 -> mcasp3 */ | |
| 2608 | +static struct omap_hwmod_ocp_if dra7xx_l4_per2__mcasp3 = { | |
| 2609 | + .master = &dra7xx_l4_per2_hwmod, | |
| 2610 | + .slave = &dra7xx_mcasp3_hwmod, | |
| 2611 | + .clk = "l4_root_clk_div", | |
| 2612 | + .user = OCP_USER_MPU | OCP_USER_SDMA, | |
| 2613 | +}; | |
| 2614 | + | |
| 2615 | +/* l3_main_1 -> mcasp3 */ | |
| 2616 | +static struct omap_hwmod_ocp_if dra7xx_l3_main_1__mcasp3 = { | |
| 2617 | + .master = &dra7xx_l3_main_1_hwmod, | |
| 2618 | + .slave = &dra7xx_mcasp3_hwmod, | |
| 2619 | + .clk = "l3_iclk_div", | |
| 2620 | + .user = OCP_USER_MPU | OCP_USER_SDMA, | |
| 2621 | +}; | |
| 2622 | + | |
| 2569 | 2623 | /* l4_per1 -> elm */ |
| 2570 | 2624 | static struct omap_hwmod_ocp_if dra7xx_l4_per1__elm = { |
| 2571 | 2625 | .master = &dra7xx_l4_per1_hwmod, |
| @@ -3308,6 +3362,8 @@ static struct omap_hwmod_ocp_if *dra7xx_hwmod_ocp_ifs[] __initdata = { · 736 unmodified lines | ||
| 3308 | 3362 | &dra7xx_l4_wkup__dcan1, |
| 3309 | 3363 | &dra7xx_l4_per2__dcan2, |
| 3310 | 3364 | &dra7xx_l4_per2__cpgmac0, |
| 3365 | + &dra7xx_l4_per2__mcasp3, | |
| 3366 | + &dra7xx_l3_main_1__mcasp3, | |
| 3311 | 3367 | &dra7xx_gmac__mdio, |
| 3312 | 3368 | &dra7xx_l4_cfg__dma_system, |
| 3313 | 3369 | &dra7xx_l3_main_1__dss, |
| @@ -144,6 +144,7 @@ static struct omap_hwmod dm81xx_l4_ls_hwmod = { · 143 unmodified lines | ||
| 144 | 144 | .name = "l4_ls", |
| 145 | 145 | .clkdm_name = "alwon_l3s_clkdm", |
| 146 | 146 | .class = &l4_hwmod_class, |
| 147 | + .flags = HWMOD_NO_IDLEST, | |
| 147 | 148 | }; |
| 148 | 149 | |
| 149 | 150 | /* |
| @@ -155,6 +156,7 @@ static struct omap_hwmod dm81xx_l4_hs_hwmod = { · 5 unmodified lines | ||
| 155 | 156 | .name = "l4_hs", |
| 156 | 157 | .clkdm_name = "alwon_l3_med_clkdm", |
| 157 | 158 | .class = &l4_hwmod_class, |
| 159 | + .flags = HWMOD_NO_IDLEST, | |
| 158 | 160 | }; |
| 159 | 161 | |
| 160 | 162 | /* L3 slow -> L4 ls peripheral interface running at 125MHz */ |
| @@ -850,6 +852,7 @@ static struct omap_hwmod dm816x_emac0_hwmod = { · 689 unmodified lines | ||
| 850 | 852 | .name = "emac0", |
| 851 | 853 | .clkdm_name = "alwon_ethernet_clkdm", |
| 852 | 854 | .class = &dm816x_emac_hwmod_class, |
| 855 | + .flags = HWMOD_NO_IDLEST, | |
| 853 | 856 | }; |
| 854 | 857 | |
| 855 | 858 | static struct omap_hwmod_ocp_if dm81xx_l4_hs__emac0 = { |
| @@ -24,9 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <linux/platform_data/iommu-omap.h> |
| 25 | 25 | #include <linux/platform_data/wkup_m3.h> |
| 26 | 26 | |
| 27 | −#include <asm/siginfo.h> | |
| 28 | −#include <asm/signal.h> | |
| 29 | − | |
| 30 | 27 | #include "common.h" |
| 31 | 28 | #include "common-board-devices.h" |
| 32 | 29 | #include "dss-common.h" |
| @@ -385,29 +382,6 @@ static void __init omap3_pandora_legacy_init(void) · 352 unmodified lines | ||
| 385 | 382 | } |
| 386 | 383 | #endif /* CONFIG_ARCH_OMAP3 */ |
| 387 | 384 | |
| 388 | −#ifdef CONFIG_SOC_TI81XX | |
| 389 | −static int fault_fixed_up; | |
| 390 | − | |
| 391 | −static int t410_abort_handler(unsigned long addr, unsigned int fsr, | |
| 392 | − struct pt_regs *regs) | |
| 393 | −{ | |
| 394 | − if ((fsr == 0x406 || fsr == 0xc06) && !fault_fixed_up) { | |
| 395 | − pr_warn("External imprecise Data abort at addr=%#lx, fsr=%#x ignored.\n", | |
| 396 | − addr, fsr); | |
| 397 | − fault_fixed_up = 1; | |
| 398 | − return 0; | |
| 399 | − } | |
| 400 | − | |
| 401 | − return 1; | |
| 402 | −} | |
| 403 | − | |
| 404 | −static void __init t410_abort_init(void) | |
| 405 | −{ | |
| 406 | − hook_fault_code(16 + 6, t410_abort_handler, SIGBUS, BUS_OBJERR, | |
| 407 | − "imprecise external abort"); | |
| 408 | −} | |
| 409 | −#endif | |
| 410 | − | |
| 411 | 385 | #if defined(CONFIG_ARCH_OMAP4) || defined(CONFIG_SOC_OMAP5) |
| 412 | 386 | static struct iommu_platform_data omap4_iommu_pdata = { |
| 413 | 387 | .reset_name = "mmu_cache", |
| @@ -536,9 +510,6 @@ static struct pdata_init pdata_quirks[] __initdata = { · 122 unmodified lines | ||
| 536 | 510 | { "openpandora,omap3-pandora-600mhz", omap3_pandora_legacy_init, }, |
| 537 | 511 | { "openpandora,omap3-pandora-1ghz", omap3_pandora_legacy_init, }, |
| 538 | 512 | #endif |
| 539 | −#ifdef CONFIG_SOC_TI81XX | |
| 540 | − { "hp,t410", t410_abort_init, }, | |
| 541 | −#endif | |
| 542 | 513 | #ifdef CONFIG_SOC_OMAP5 |
| 543 | 514 | { "ti,omap5-uevm", omap5_uevm_legacy_init, }, |
| 544 | 515 | #endif |
| @@ -301,11 +301,11 @@ static void omap3_pm_idle(void) · 300 unmodified lines | ||
| 301 | 301 | if (omap_irq_pending()) |
| 302 | 302 | return; |
| 303 | 303 | |
| 304 | − trace_cpu_idle(1, smp_processor_id()); | |
| 304 | + trace_cpu_idle_rcuidle(1, smp_processor_id()); | |
| 305 | 305 | |
| 306 | 306 | omap_sram_idle(); |
| 307 | 307 | |
| 308 | − trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); | |
| 308 | + trace_cpu_idle_rcuidle(PWR_EVENT_EXIT, smp_processor_id()); | |
| 309 | 309 | } |
| 310 | 310 | |
| 311 | 311 | #ifdef CONFIG_SUSPEND |
| @@ -21,5 +21,5 @@ · 20 unmodified lines | ||
| 21 | 21 | @ find cause bits that are unmasked |
| 22 | 22 | ands \irqstat, \irqstat, \tmp @ clear Z flag if any |
| 23 | 23 | clzne \irqnr, \irqstat @ calc irqnr |
| 24 | − rsbne \irqnr, \irqnr, #31 | |
| 24 | + rsbne \irqnr, \irqnr, #32 | |
| 25 | 25 | .endm |
| @@ -889,6 +889,7 @@ static void __init e680_init(void) · 888 unmodified lines | ||
| 889 | 889 | |
| 890 | 890 | pxa_set_keypad_info(&e680_keypad_platform_data); |
| 891 | 891 | |
| 892 | + pwm_add_table(ezx_pwm_lookup, ARRAY_SIZE(ezx_pwm_lookup)); | |
| 892 | 893 | platform_add_devices(ARRAY_AND_SIZE(ezx_devices)); |
| 893 | 894 | platform_add_devices(ARRAY_AND_SIZE(e680_devices)); |
| 894 | 895 | } |
| @@ -956,6 +957,7 @@ static void __init a1200_init(void) · 61 unmodified lines | ||
| 956 | 957 | |
| 957 | 958 | pxa_set_keypad_info(&a1200_keypad_platform_data); |
| 958 | 959 | |
| 960 | + pwm_add_table(ezx_pwm_lookup, ARRAY_SIZE(ezx_pwm_lookup)); | |
| 959 | 961 | platform_add_devices(ARRAY_AND_SIZE(ezx_devices)); |
| 960 | 962 | platform_add_devices(ARRAY_AND_SIZE(a1200_devices)); |
| 961 | 963 | } |
| @@ -1148,6 +1150,7 @@ static void __init a910_init(void) · 186 unmodified lines | ||
| 1148 | 1150 | platform_device_register(&a910_camera); |
| 1149 | 1151 | } |
| 1150 | 1152 | |
| 1153 | + pwm_add_table(ezx_pwm_lookup, ARRAY_SIZE(ezx_pwm_lookup)); | |
| 1151 | 1154 | platform_add_devices(ARRAY_AND_SIZE(ezx_devices)); |
| 1152 | 1155 | platform_add_devices(ARRAY_AND_SIZE(a910_devices)); |
| 1153 | 1156 | } |
| @@ -1215,6 +1218,7 @@ static void __init e6_init(void) · 61 unmodified lines | ||
| 1215 | 1218 | |
| 1216 | 1219 | pxa_set_keypad_info(&e6_keypad_platform_data); |
| 1217 | 1220 | |
| 1221 | + pwm_add_table(ezx_pwm_lookup, ARRAY_SIZE(ezx_pwm_lookup)); | |
| 1218 | 1222 | platform_add_devices(ARRAY_AND_SIZE(ezx_devices)); |
| 1219 | 1223 | platform_add_devices(ARRAY_AND_SIZE(e6_devices)); |
| 1220 | 1224 | } |
| @@ -1256,6 +1260,7 @@ static void __init e2_init(void) · 35 unmodified lines | ||
| 1256 | 1260 | |
| 1257 | 1261 | pxa_set_keypad_info(&e2_keypad_platform_data); |
| 1258 | 1262 | |
| 1263 | + pwm_add_table(ezx_pwm_lookup, ARRAY_SIZE(ezx_pwm_lookup)); | |
| 1259 | 1264 | platform_add_devices(ARRAY_AND_SIZE(ezx_devices)); |
| 1260 | 1265 | platform_add_devices(ARRAY_AND_SIZE(e2_devices)); |
| 1261 | 1266 | } |
| @@ -344,7 +344,7 @@ void __init palm27x_pwm_init(int bl, int lcd) · 343 unmodified lines | ||
| 344 | 344 | { |
| 345 | 345 | palm_bl_power = bl; |
| 346 | 346 | palm_lcd_power = lcd; |
| 347 | − pwm_add_lookup(palm27x_pwm_lookup, ARRAY_SIZE(palm27x_pwm_lookup)); | |
| 347 | + pwm_add_table(palm27x_pwm_lookup, ARRAY_SIZE(palm27x_pwm_lookup)); | |
| 348 | 348 | platform_device_register(&palm27x_backlight); |
| 349 | 349 | } |
| 350 | 350 | #endif |
| @@ -169,7 +169,7 @@ static inline void palmtc_keys_init(void) {} · 168 unmodified lines | ||
| 169 | 169 | #if defined(CONFIG_BACKLIGHT_PWM) || defined(CONFIG_BACKLIGHT_PWM_MODULE) |
| 170 | 170 | static struct pwm_lookup palmtc_pwm_lookup[] = { |
| 171 | 171 | PWM_LOOKUP("pxa25x-pwm.1", 0, "pwm-backlight.0", NULL, PALMTC_PERIOD_NS, |
| 172 | − PWM_PERIOD_NORMAL), | |
| 172 | + PWM_POLARITY_NORMAL), | |
| 173 | 173 | }; |
| 174 | 174 | |
| 175 | 175 | static struct platform_pwm_backlight_data palmtc_backlight_data = { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <plat/cpu.h> |
| 21 | 21 | #include <plat/cpu-freq-core.h> |
| 22 | 22 | |
| 23 | −static struct cpufreq_frequency_table s3c2440_plls_12[] __initdata = { | |
| 23 | +static struct cpufreq_frequency_table s3c2440_plls_12[] = { | |
| 24 | 24 | { .frequency = 75000000, .driver_data = PLLVAL(0x75, 3, 3), }, /* FVco 600.000000 */ |
| 25 | 25 | { .frequency = 80000000, .driver_data = PLLVAL(0x98, 4, 3), }, /* FVco 640.000000 */ |
| 26 | 26 | { .frequency = 90000000, .driver_data = PLLVAL(0x70, 2, 3), }, /* FVco 720.000000 */ |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <plat/cpu.h> |
| 21 | 21 | #include <plat/cpu-freq-core.h> |
| 22 | 22 | |
| 23 | −static struct cpufreq_frequency_table s3c2440_plls_169344[] __initdata = { | |
| 23 | +static struct cpufreq_frequency_table s3c2440_plls_169344[] = { | |
| 24 | 24 | { .frequency = 78019200, .driver_data = PLLVAL(121, 5, 3), }, /* FVco 624.153600 */ |
| 25 | 25 | { .frequency = 84067200, .driver_data = PLLVAL(131, 5, 3), }, /* FVco 672.537600 */ |
| 26 | 26 | { .frequency = 90115200, .driver_data = PLLVAL(141, 5, 3), }, /* FVco 720.921600 */ |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include "common.h" |
| 20 | 20 | #include "rcar-gen2.h" |
| 21 | 21 | |
| 22 | −static const char *r8a7793_boards_compat_dt[] __initconst = { | |
| 22 | +static const char * const r8a7793_boards_compat_dt[] __initconst = { | |
| 23 | 23 | "renesas,r8a7793", |
| 24 | 24 | NULL, |
| 25 | 25 | }; |
| @@ -13,7 +13,7 @@ config SOC_ZX296702 · 12 unmodified lines | ||
| 13 | 13 | select ARM_GLOBAL_TIMER |
| 14 | 14 | select HAVE_ARM_SCU if SMP |
| 15 | 15 | select HAVE_ARM_TWD if SMP |
| 16 | − select PM_GENERIC_DOMAINS | |
| 16 | + select PM_GENERIC_DOMAINS if PM | |
| 17 | 17 | help |
| 18 | 18 | Support for ZTE ZX296702 SoC which is a dual core CortexA9MP |
| 19 | 19 | endif |
| @@ -49,7 +49,7 @@ config ARM64 · 48 unmodified lines | ||
| 49 | 49 | select HAVE_ARCH_AUDITSYSCALL |
| 50 | 50 | select HAVE_ARCH_BITREVERSE |
| 51 | 51 | select HAVE_ARCH_JUMP_LABEL |
| 52 | − select HAVE_ARCH_KASAN if SPARSEMEM_VMEMMAP | |
| 52 | + select HAVE_ARCH_KASAN if SPARSEMEM_VMEMMAP && !(ARM64_16K_PAGES && ARM64_VA_BITS_48) | |
| 53 | 53 | select HAVE_ARCH_KGDB |
| 54 | 54 | select HAVE_ARCH_SECCOMP_FILTER |
| 55 | 55 | select HAVE_ARCH_TRACEHOOK |
| @@ -269,6 +269,7 @@ · 268 unmodified lines | ||
| 269 | 269 | clock-frequency = <0>; /* Updated by bootloader */ |
| 270 | 270 | voltage-ranges = <1800 1800 3300 3300>; |
| 271 | 271 | sdhci,auto-cmd12; |
| 272 | + little-endian; | |
| 272 | 273 | bus-width = <4>; |
| 273 | 274 | }; |
| 274 | 275 | |
| @@ -277,6 +278,7 @@ · 2 unmodified lines | ||
| 277 | 278 | reg = <0x0 0x2300000 0x0 0x10000>; |
| 278 | 279 | interrupts = <0 36 0x4>; /* Level high type */ |
| 279 | 280 | gpio-controller; |
| 281 | + little-endian; | |
| 280 | 282 | #gpio-cells = <2>; |
| 281 | 283 | interrupt-controller; |
| 282 | 284 | #interrupt-cells = <2>; |
| @@ -287,6 +289,7 @@ · 4 unmodified lines | ||
| 287 | 289 | reg = <0x0 0x2310000 0x0 0x10000>; |
| 288 | 290 | interrupts = <0 36 0x4>; /* Level high type */ |
| 289 | 291 | gpio-controller; |
| 292 | + little-endian; | |
| 290 | 293 | #gpio-cells = <2>; |
| 291 | 294 | interrupt-controller; |
| 292 | 295 | #interrupt-cells = <2>; |
| @@ -297,6 +300,7 @@ · 4 unmodified lines | ||
| 297 | 300 | reg = <0x0 0x2320000 0x0 0x10000>; |
| 298 | 301 | interrupts = <0 37 0x4>; /* Level high type */ |
| 299 | 302 | gpio-controller; |
| 303 | + little-endian; | |
| 300 | 304 | #gpio-cells = <2>; |
| 301 | 305 | interrupt-controller; |
| 302 | 306 | #interrupt-cells = <2>; |
| @@ -307,6 +311,7 @@ · 4 unmodified lines | ||
| 307 | 311 | reg = <0x0 0x2330000 0x0 0x10000>; |
| 308 | 312 | interrupts = <0 37 0x4>; /* Level high type */ |
| 309 | 313 | gpio-controller; |
| 314 | + little-endian; | |
| 310 | 315 | #gpio-cells = <2>; |
| 311 | 316 | interrupt-controller; |
| 312 | 317 | #interrupt-cells = <2>; |
| @@ -77,6 +77,7 @@ · 76 unmodified lines | ||
| 77 | 77 | #ifndef __ASSEMBLY__ |
| 78 | 78 | |
| 79 | 79 | #include <linux/stringify.h> |
| 80 | +#include <asm/barrier.h> | |
| 80 | 81 | |
| 81 | 82 | /* |
| 82 | 83 | * Low-level accessors |
| @@ -47,8 +47,12 @@ enum ftr_type { · 46 unmodified lines | ||
| 47 | 47 | #define FTR_STRICT true /* SANITY check strict matching required */ |
| 48 | 48 | #define FTR_NONSTRICT false /* SANITY check ignored */ |
| 49 | 49 | |
| 50 | +#define FTR_SIGNED true /* Value should be treated as signed */ | |
| 51 | +#define FTR_UNSIGNED false /* Value should be treated as unsigned */ | |
| 52 | + | |
| 50 | 53 | struct arm64_ftr_bits { |
| 51 | − bool strict; /* CPU Sanity check: strict matching required ? */ | |
| 54 | + bool sign; /* Value is signed ? */ | |
| 55 | + bool strict; /* CPU Sanity check: strict matching required ? */ | |
| 52 | 56 | enum ftr_type type; |
| 53 | 57 | u8 shift; |
| 54 | 58 | u8 width; |
| @@ -124,6 +128,18 @@ cpuid_feature_extract_field(u64 features, int field) · 69 unmodified lines | ||
| 124 | 128 | return cpuid_feature_extract_field_width(features, field, 4); |
| 125 | 129 | } |
| 126 | 130 | |
| 131 | +static inline unsigned int __attribute_const__ | |
| 132 | +cpuid_feature_extract_unsigned_field_width(u64 features, int field, int width) | |
| 133 | +{ | |
| 134 | + return (u64)(features << (64 - width - field)) >> (64 - width); | |
| 135 | +} | |
| 136 | + | |
| 137 | +static inline unsigned int __attribute_const__ | |
| 138 | +cpuid_feature_extract_unsigned_field(u64 features, int field) | |
| 139 | +{ | |
| 140 | + return cpuid_feature_extract_unsigned_field_width(features, field, 4); | |
| 141 | +} | |
| 142 | + | |
| 127 | 143 | static inline u64 arm64_ftr_mask(struct arm64_ftr_bits *ftrp) |
| 128 | 144 | { |
| 129 | 145 | return (u64)GENMASK(ftrp->shift + ftrp->width - 1, ftrp->shift); |
| @@ -131,7 +147,9 @@ static inline u64 arm64_ftr_mask(struct arm64_ftr_bits *ftrp) · 1 unmodified line | ||
| 131 | 147 | |
| 132 | 148 | static inline s64 arm64_ftr_value(struct arm64_ftr_bits *ftrp, u64 val) |
| 133 | 149 | { |
| 134 | − return cpuid_feature_extract_field_width(val, ftrp->shift, ftrp->width); | |
| 150 | + return ftrp->sign ? | |
| 151 | + cpuid_feature_extract_field_width(val, ftrp->shift, ftrp->width) : | |
| 152 | + cpuid_feature_extract_unsigned_field_width(val, ftrp->shift, ftrp->width); | |
| 135 | 153 | } |
| 136 | 154 | |
| 137 | 155 | static inline bool id_aa64mmfr0_mixed_endian_el0(u64 mmfr0) |
| @@ -138,16 +138,18 @@ extern struct pmu perf_ops_bp; · 137 unmodified lines | ||
| 138 | 138 | /* Determine number of BRP registers available. */ |
| 139 | 139 | static inline int get_num_brps(void) |
| 140 | 140 | { |
| 141 | + u64 dfr0 = read_system_reg(SYS_ID_AA64DFR0_EL1); | |
| 141 | 142 | return 1 + |
| 142 | − cpuid_feature_extract_field(read_system_reg(SYS_ID_AA64DFR0_EL1), | |
| 143 | + cpuid_feature_extract_unsigned_field(dfr0, | |
| 143 | 144 | ID_AA64DFR0_BRPS_SHIFT); |
| 144 | 145 | } |
| 145 | 146 | |
| 146 | 147 | /* Determine number of WRP registers available. */ |
| 147 | 148 | static inline int get_num_wrps(void) |
| 148 | 149 | { |
| 150 | + u64 dfr0 = read_system_reg(SYS_ID_AA64DFR0_EL1); | |
| 149 | 151 | return 1 + |
| 150 | − cpuid_feature_extract_field(read_system_reg(SYS_ID_AA64DFR0_EL1), | |
| 152 | + cpuid_feature_extract_unsigned_field(dfr0, | |
| 151 | 153 | ID_AA64DFR0_WRPS_SHIFT); |
| 152 | 154 | } |
| 153 | 155 | |
| @@ -7,4 +7,9 @@ struct pt_regs; · 6 unmodified lines | ||
| 7 | 7 | |
| 8 | 8 | extern void set_handle_irq(void (*handle_irq)(struct pt_regs *)); |
| 9 | 9 | |
| 10 | +static inline int nr_legacy_irqs(void) | |
| 11 | +{ | |
| 12 | + return 0; | |
| 13 | +} | |
| 14 | + | |
| 10 | 15 | #endif |
| @@ -125,6 +125,7 @@ · 124 unmodified lines | ||
| 125 | 125 | #define VTCR_EL2_SL0_LVL1 (1 << 6) |
| 126 | 126 | #define VTCR_EL2_T0SZ_MASK 0x3f |
| 127 | 127 | #define VTCR_EL2_T0SZ_40B 24 |
| 128 | +#define VTCR_EL2_VS 19 | |
| 128 | 129 | |
| 129 | 130 | /* |
| 130 | 131 | * We configure the Stage-2 page tables to always restrict the IPA space to be |
| @@ -169,7 +170,7 @@ · 38 unmodified lines | ||
| 169 | 170 | #define VTTBR_BADDR_SHIFT (VTTBR_X - 1) |
| 170 | 171 | #define VTTBR_BADDR_MASK (((UL(1) << (PHYS_MASK_SHIFT - VTTBR_X)) - 1) << VTTBR_BADDR_SHIFT) |
| 171 | 172 | #define VTTBR_VMID_SHIFT (UL(48)) |
| 172 | −#define VTTBR_VMID_MASK (UL(0xFF) << VTTBR_VMID_SHIFT) | |
| 173 | +#define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) | |
| 173 | 174 | |
| 174 | 175 | /* Hyp System Trap Register */ |
| 175 | 176 | #define HSTR_EL2_T(x) (1 << x) |
| @@ -20,82 +20,6 @@ · 19 unmodified lines | ||
| 20 | 20 | |
| 21 | 21 | #include <asm/virt.h> |
| 22 | 22 | |
| 23 | −/* | |
| 24 | − * 0 is reserved as an invalid value. | |
| 25 | − * Order *must* be kept in sync with the hyp switch code. | |
| 26 | − */ | |
| 27 | −#define MPIDR_EL1 1 /* MultiProcessor Affinity Register */ | |
| 28 | −#define CSSELR_EL1 2 /* Cache Size Selection Register */ | |
| 29 | −#define SCTLR_EL1 3 /* System Control Register */ | |
| 30 | −#define ACTLR_EL1 4 /* Auxiliary Control Register */ | |
| 31 | −#define CPACR_EL1 5 /* Coprocessor Access Control */ | |
| 32 | −#define TTBR0_EL1 6 /* Translation Table Base Register 0 */ | |
| 33 | −#define TTBR1_EL1 7 /* Translation Table Base Register 1 */ | |
| 34 | −#define TCR_EL1 8 /* Translation Control Register */ | |
| 35 | −#define ESR_EL1 9 /* Exception Syndrome Register */ | |
| 36 | −#define AFSR0_EL1 10 /* Auxilary Fault Status Register 0 */ | |
| 37 | −#define AFSR1_EL1 11 /* Auxilary Fault Status Register 1 */ | |
| 38 | −#define FAR_EL1 12 /* Fault Address Register */ | |
| 39 | −#define MAIR_EL1 13 /* Memory Attribute Indirection Register */ | |
| 40 | −#define VBAR_EL1 14 /* Vector Base Address Register */ | |
| 41 | −#define CONTEXTIDR_EL1 15 /* Context ID Register */ | |
| 42 | −#define TPIDR_EL0 16 /* Thread ID, User R/W */ | |
| 43 | −#define TPIDRRO_EL0 17 /* Thread ID, User R/O */ | |
| 44 | −#define TPIDR_EL1 18 /* Thread ID, Privileged */ | |
| 45 | −#define AMAIR_EL1 19 /* Aux Memory Attribute Indirection Register */ | |
| 46 | −#define CNTKCTL_EL1 20 /* Timer Control Register (EL1) */ | |
| 47 | −#define PAR_EL1 21 /* Physical Address Register */ | |
| 48 | −#define MDSCR_EL1 22 /* Monitor Debug System Control Register */ | |
| 49 | −#define MDCCINT_EL1 23 /* Monitor Debug Comms Channel Interrupt Enable Reg */ | |
| 50 | − | |
| 51 | −/* 32bit specific registers. Keep them at the end of the range */ | |
| 52 | −#define DACR32_EL2 24 /* Domain Access Control Register */ | |
| 53 | −#define IFSR32_EL2 25 /* Instruction Fault Status Register */ | |
| 54 | −#define FPEXC32_EL2 26 /* Floating-Point Exception Control Register */ | |
| 55 | −#define DBGVCR32_EL2 27 /* Debug Vector Catch Register */ | |
| 56 | −#define NR_SYS_REGS 28 | |
| 57 | − | |
| 58 | −/* 32bit mapping */ | |
| 59 | −#define c0_MPIDR (MPIDR_EL1 * 2) /* MultiProcessor ID Register */ | |
| 60 | −#define c0_CSSELR (CSSELR_EL1 * 2)/* Cache Size Selection Register */ | |
| 61 | −#define c1_SCTLR (SCTLR_EL1 * 2) /* System Control Register */ | |
| 62 | −#define c1_ACTLR (ACTLR_EL1 * 2) /* Auxiliary Control Register */ | |
| 63 | −#define c1_CPACR (CPACR_EL1 * 2) /* Coprocessor Access Control */ | |
| 64 | −#define c2_TTBR0 (TTBR0_EL1 * 2) /* Translation Table Base Register 0 */ | |
| 65 | −#define c2_TTBR0_high (c2_TTBR0 + 1) /* TTBR0 top 32 bits */ | |
| 66 | −#define c2_TTBR1 (TTBR1_EL1 * 2) /* Translation Table Base Register 1 */ | |
| 67 | −#define c2_TTBR1_high (c2_TTBR1 + 1) /* TTBR1 top 32 bits */ | |
| 68 | −#define c2_TTBCR (TCR_EL1 * 2) /* Translation Table Base Control R. */ | |
| 69 | −#define c3_DACR (DACR32_EL2 * 2)/* Domain Access Control Register */ | |
| 70 | −#define c5_DFSR (ESR_EL1 * 2) /* Data Fault Status Register */ | |
| 71 | −#define c5_IFSR (IFSR32_EL2 * 2)/* Instruction Fault Status Register */ | |
| 72 | −#define c5_ADFSR (AFSR0_EL1 * 2) /* Auxiliary Data Fault Status R */ | |
| 73 | −#define c5_AIFSR (AFSR1_EL1 * 2) /* Auxiliary Instr Fault Status R */ | |
| 74 | −#define c6_DFAR (FAR_EL1 * 2) /* Data Fault Address Register */ | |
| 75 | −#define c6_IFAR (c6_DFAR + 1) /* Instruction Fault Address Register */ | |
| 76 | −#define c7_PAR (PAR_EL1 * 2) /* Physical Address Register */ | |
| 77 | −#define c7_PAR_high (c7_PAR + 1) /* PAR top 32 bits */ | |
| 78 | −#define c10_PRRR (MAIR_EL1 * 2) /* Primary Region Remap Register */ | |
| 79 | −#define c10_NMRR (c10_PRRR + 1) /* Normal Memory Remap Register */ | |
| 80 | −#define c12_VBAR (VBAR_EL1 * 2) /* Vector Base Address Register */ | |
| 81 | −#define c13_CID (CONTEXTIDR_EL1 * 2) /* Context ID Register */ | |
| 82 | −#define c13_TID_URW (TPIDR_EL0 * 2) /* Thread ID, User R/W */ | |
| 83 | −#define c13_TID_URO (TPIDRRO_EL0 * 2)/* Thread ID, User R/O */ | |
| 84 | −#define c13_TID_PRIV (TPIDR_EL1 * 2) /* Thread ID, Privileged */ | |
| 85 | −#define c10_AMAIR0 (AMAIR_EL1 * 2) /* Aux Memory Attr Indirection Reg */ | |
| 86 | −#define c10_AMAIR1 (c10_AMAIR0 + 1)/* Aux Memory Attr Indirection Reg */ | |
| 87 | −#define c14_CNTKCTL (CNTKCTL_EL1 * 2) /* Timer Control Register (PL1) */ | |
| 88 | − | |
| 89 | −#define cp14_DBGDSCRext (MDSCR_EL1 * 2) | |
| 90 | −#define cp14_DBGBCR0 (DBGBCR0_EL1 * 2) | |
| 91 | −#define cp14_DBGBVR0 (DBGBVR0_EL1 * 2) | |
| 92 | −#define cp14_DBGBXVR0 (cp14_DBGBVR0 + 1) | |
| 93 | −#define cp14_DBGWCR0 (DBGWCR0_EL1 * 2) | |
| 94 | −#define cp14_DBGWVR0 (DBGWVR0_EL1 * 2) | |
| 95 | −#define cp14_DBGDCCINT (MDCCINT_EL1 * 2) | |
| 96 | − | |
| 97 | −#define NR_COPRO_REGS (NR_SYS_REGS * 2) | |
| 98 | − | |
| 99 | 23 | #define ARM_EXCEPTION_IRQ 0 |
| 100 | 24 | #define ARM_EXCEPTION_TRAP 1 |
| 101 | 25 | |
| @@ -26,7 +26,6 @@ · 25 unmodified lines | ||
| 26 | 26 | |
| 27 | 27 | #include <asm/esr.h> |
| 28 | 28 | #include <asm/kvm_arm.h> |
| 29 | −#include <asm/kvm_asm.h> | |
| 30 | 29 | #include <asm/kvm_mmio.h> |
| 31 | 30 | #include <asm/ptrace.h> |
| 32 | 31 | #include <asm/cputype.h> |
| @@ -100,13 +99,21 @@ static inline void vcpu_set_thumb(struct kvm_vcpu *vcpu) · 67 unmodified lines | ||
| 100 | 99 | } |
| 101 | 100 | |
| 102 | 101 | /* |
| 103 | − * vcpu_reg should always be passed a register number coming from a | |
| 104 | − * read of ESR_EL2. Otherwise, it may give the wrong result on AArch32 | |
| 105 | − * with banked registers. | |
| 102 | + * vcpu_get_reg and vcpu_set_reg should always be passed a register number | |
| 103 | + * coming from a read of ESR_EL2. Otherwise, it may give the wrong result on | |
| 104 | + * AArch32 with banked registers. | |
| 106 | 105 | */ |
| 107 | −static inline unsigned long *vcpu_reg(const struct kvm_vcpu *vcpu, u8 reg_num) | |
| 106 | +static inline unsigned long vcpu_get_reg(const struct kvm_vcpu *vcpu, | |
| 107 | + u8 reg_num) | |
| 108 | 108 | { |
| 109 | − return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.regs[reg_num]; | |
| 109 | + return (reg_num == 31) ? 0 : vcpu_gp_regs(vcpu)->regs.regs[reg_num]; | |
| 110 | +} | |
| 111 | + | |
| 112 | +static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, | |
| 113 | + unsigned long val) | |
| 114 | +{ | |
| 115 | + if (reg_num != 31) | |
| 116 | + vcpu_gp_regs(vcpu)->regs.regs[reg_num] = val; | |
| 110 | 117 | } |
| 111 | 118 | |
| 112 | 119 | /* Get vcpu SPSR for current mode */ |
| @@ -25,7 +25,6 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <linux/types.h> |
| 26 | 26 | #include <linux/kvm_types.h> |
| 27 | 27 | #include <asm/kvm.h> |
| 28 | −#include <asm/kvm_asm.h> | |
| 29 | 28 | #include <asm/kvm_mmio.h> |
| 30 | 29 | |
| 31 | 30 | #define __KVM_HAVE_ARCH_INTC_INITIALIZED |
| @@ -85,6 +84,86 @@ struct kvm_vcpu_fault_info { · 53 unmodified lines | ||
| 85 | 84 | u64 hpfar_el2; /* Hyp IPA Fault Address Register */ |
| 86 | 85 | }; |
| 87 | 86 | |
| 87 | +/* | |
| 88 | + * 0 is reserved as an invalid value. | |
| 89 | + * Order should be kept in sync with the save/restore code. | |
| 90 | + */ | |
| 91 | +enum vcpu_sysreg { | |
| 92 | + __INVALID_SYSREG__, | |
| 93 | + MPIDR_EL1, /* MultiProcessor Affinity Register */ | |
| 94 | + CSSELR_EL1, /* Cache Size Selection Register */ | |
| 95 | + SCTLR_EL1, /* System Control Register */ | |
| 96 | + ACTLR_EL1, /* Auxiliary Control Register */ | |
| 97 | + CPACR_EL1, /* Coprocessor Access Control */ | |
| 98 | + TTBR0_EL1, /* Translation Table Base Register 0 */ | |
| 99 | + TTBR1_EL1, /* Translation Table Base Register 1 */ | |
| 100 | + TCR_EL1, /* Translation Control Register */ | |
| 101 | + ESR_EL1, /* Exception Syndrome Register */ | |
| 102 | + AFSR0_EL1, /* Auxilary Fault Status Register 0 */ | |
| 103 | + AFSR1_EL1, /* Auxilary Fault Status Register 1 */ | |
| 104 | + FAR_EL1, /* Fault Address Register */ | |
| 105 | + MAIR_EL1, /* Memory Attribute Indirection Register */ | |
| 106 | + VBAR_EL1, /* Vector Base Address Register */ | |
| 107 | + CONTEXTIDR_EL1, /* Context ID Register */ | |
| 108 | + TPIDR_EL0, /* Thread ID, User R/W */ | |
| 109 | + TPIDRRO_EL0, /* Thread ID, User R/O */ | |
| 110 | + TPIDR_EL1, /* Thread ID, Privileged */ | |
| 111 | + AMAIR_EL1, /* Aux Memory Attribute Indirection Register */ | |
| 112 | + CNTKCTL_EL1, /* Timer Control Register (EL1) */ | |
| 113 | + PAR_EL1, /* Physical Address Register */ | |
| 114 | + MDSCR_EL1, /* Monitor Debug System Control Register */ | |
| 115 | + MDCCINT_EL1, /* Monitor Debug Comms Channel Interrupt Enable Reg */ | |
| 116 | + | |
| 117 | + /* 32bit specific registers. Keep them at the end of the range */ | |
| 118 | + DACR32_EL2, /* Domain Access Control Register */ | |
| 119 | + IFSR32_EL2, /* Instruction Fault Status Register */ | |
| 120 | + FPEXC32_EL2, /* Floating-Point Exception Control Register */ | |
| 121 | + DBGVCR32_EL2, /* Debug Vector Catch Register */ | |
| 122 | + | |
| 123 | + NR_SYS_REGS /* Nothing after this line! */ | |
| 124 | +}; | |
| 125 | + | |
| 126 | +/* 32bit mapping */ | |
| 127 | +#define c0_MPIDR (MPIDR_EL1 * 2) /* MultiProcessor ID Register */ | |
| 128 | +#define c0_CSSELR (CSSELR_EL1 * 2)/* Cache Size Selection Register */ | |
| 129 | +#define c1_SCTLR (SCTLR_EL1 * 2) /* System Control Register */ | |
| 130 | +#define c1_ACTLR (ACTLR_EL1 * 2) /* Auxiliary Control Register */ | |
| 131 | +#define c1_CPACR (CPACR_EL1 * 2) /* Coprocessor Access Control */ | |
| 132 | +#define c2_TTBR0 (TTBR0_EL1 * 2) /* Translation Table Base Register 0 */ | |
| 133 | +#define c2_TTBR0_high (c2_TTBR0 + 1) /* TTBR0 top 32 bits */ | |
| 134 | +#define c2_TTBR1 (TTBR1_EL1 * 2) /* Translation Table Base Register 1 */ | |
| 135 | +#define c2_TTBR1_high (c2_TTBR1 + 1) /* TTBR1 top 32 bits */ | |
| 136 | +#define c2_TTBCR (TCR_EL1 * 2) /* Translation Table Base Control R. */ | |
| 137 | +#define c3_DACR (DACR32_EL2 * 2)/* Domain Access Control Register */ | |
| 138 | +#define c5_DFSR (ESR_EL1 * 2) /* Data Fault Status Register */ | |
| 139 | +#define c5_IFSR (IFSR32_EL2 * 2)/* Instruction Fault Status Register */ | |
| 140 | +#define c5_ADFSR (AFSR0_EL1 * 2) /* Auxiliary Data Fault Status R */ | |
| 141 | +#define c5_AIFSR (AFSR1_EL1 * 2) /* Auxiliary Instr Fault Status R */ | |
| 142 | +#define c6_DFAR (FAR_EL1 * 2) /* Data Fault Address Register */ | |
| 143 | +#define c6_IFAR (c6_DFAR + 1) /* Instruction Fault Address Register */ | |
| 144 | +#define c7_PAR (PAR_EL1 * 2) /* Physical Address Register */ | |
| 145 | +#define c7_PAR_high (c7_PAR + 1) /* PAR top 32 bits */ | |
| 146 | +#define c10_PRRR (MAIR_EL1 * 2) /* Primary Region Remap Register */ | |
| 147 | +#define c10_NMRR (c10_PRRR + 1) /* Normal Memory Remap Register */ | |
| 148 | +#define c12_VBAR (VBAR_EL1 * 2) /* Vector Base Address Register */ | |
| 149 | +#define c13_CID (CONTEXTIDR_EL1 * 2) /* Context ID Register */ | |
| 150 | +#define c13_TID_URW (TPIDR_EL0 * 2) /* Thread ID, User R/W */ | |
| 151 | +#define c13_TID_URO (TPIDRRO_EL0 * 2)/* Thread ID, User R/O */ | |
| 152 | +#define c13_TID_PRIV (TPIDR_EL1 * 2) /* Thread ID, Privileged */ | |
| 153 | +#define c10_AMAIR0 (AMAIR_EL1 * 2) /* Aux Memory Attr Indirection Reg */ | |
| 154 | +#define c10_AMAIR1 (c10_AMAIR0 + 1)/* Aux Memory Attr Indirection Reg */ | |
| 155 | +#define c14_CNTKCTL (CNTKCTL_EL1 * 2) /* Timer Control Register (PL1) */ | |
| 156 | + | |
| 157 | +#define cp14_DBGDSCRext (MDSCR_EL1 * 2) | |
| 158 | +#define cp14_DBGBCR0 (DBGBCR0_EL1 * 2) | |
| 159 | +#define cp14_DBGBVR0 (DBGBVR0_EL1 * 2) | |
| 160 | +#define cp14_DBGBXVR0 (cp14_DBGBVR0 + 1) | |
| 161 | +#define cp14_DBGWCR0 (DBGWCR0_EL1 * 2) | |
| 162 | +#define cp14_DBGWVR0 (DBGWVR0_EL1 * 2) | |
| 163 | +#define cp14_DBGDCCINT (MDCCINT_EL1 * 2) | |
| 164 | + | |
| 165 | +#define NR_COPRO_REGS (NR_SYS_REGS * 2) | |
| 166 | + | |
| 88 | 167 | struct kvm_cpu_context { |
| 89 | 168 | struct kvm_regs gp_regs; |
| 90 | 169 | union { |
| @@ -197,6 +276,12 @@ struct kvm_vcpu_stat { · 106 unmodified lines | ||
| 197 | 276 | u32 halt_successful_poll; |
| 198 | 277 | u32 halt_attempted_poll; |
| 199 | 278 | u32 halt_wakeup; |
| 279 | + u32 hvc_exit_stat; | |
| 280 | + u64 wfe_exit_stat; | |
| 281 | + u64 wfi_exit_stat; | |
| 282 | + u64 mmio_exit_user; | |
| 283 | + u64 mmio_exit_kernel; | |
| 284 | + u64 exits; | |
| 200 | 285 | }; |
| 201 | 286 | |
| 202 | 287 | int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init); |
| @@ -19,7 +19,6 @@ · 18 unmodified lines | ||
| 19 | 19 | #define __ARM64_KVM_MMIO_H__ |
| 20 | 20 | |
| 21 | 21 | #include <linux/kvm_host.h> |
| 22 | −#include <asm/kvm_asm.h> | |
| 23 | 22 | #include <asm/kvm_arm.h> |
| 24 | 23 | |
| 25 | 24 | /* |
| @@ -20,6 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | |
| 21 | 21 | #include <asm/page.h> |
| 22 | 22 | #include <asm/memory.h> |
| 23 | +#include <asm/cpufeature.h> | |
| 23 | 24 | |
| 24 | 25 | /* |
| 25 | 26 | * As we only have the TTBR0_EL2 register, we cannot express |
| @@ -158,7 +159,6 @@ static inline bool kvm_s2pmd_readonly(pmd_t *pmd) · 132 unmodified lines | ||
| 158 | 159 | #define PTRS_PER_S2_PGD_SHIFT (KVM_PHYS_SHIFT - PGDIR_SHIFT) |
| 159 | 160 | #endif |
| 160 | 161 | #define PTRS_PER_S2_PGD (1 << PTRS_PER_S2_PGD_SHIFT) |
| 161 | −#define S2_PGD_ORDER get_order(PTRS_PER_S2_PGD * sizeof(pgd_t)) | |
| 162 | 162 | |
| 163 | 163 | #define kvm_pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_S2_PGD - 1)) |
| 164 | 164 | |
| @@ -302,5 +302,12 @@ static inline void __kvm_extend_hypmap(pgd_t *boot_hyp_pgd, · 137 unmodified lines | ||
| 302 | 302 | merged_hyp_pgd[idmap_idx] = __pgd(__pa(boot_hyp_pgd) | PMD_TYPE_TABLE); |
| 303 | 303 | } |
| 304 | 304 | |
| 305 | +static inline unsigned int kvm_get_vmid_bits(void) | |
| 306 | +{ | |
| 307 | + int reg = read_system_reg(SYS_ID_AA64MMFR1_EL1); | |
| 308 | + | |
| 309 | + return (cpuid_feature_extract_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; | |
| 310 | +} | |
| 311 | + | |
| 305 | 312 | #endif /* __ASSEMBLY__ */ |
| 306 | 313 | #endif /* __ARM64_KVM_MMU_H__ */ |
| @@ -276,10 +276,14 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, · 275 unmodified lines | ||
| 276 | 276 | * hardware updates of the pte (ptep_set_access_flags safely changes |
| 277 | 277 | * valid ptes without going through an invalid entry). |
| 278 | 278 | */ |
| 279 | − if (IS_ENABLED(CONFIG_DEBUG_VM) && IS_ENABLED(CONFIG_ARM64_HW_AFDBM) && | |
| 280 | − pte_valid(*ptep)) { | |
| 281 | − BUG_ON(!pte_young(pte)); | |
| 282 | − BUG_ON(pte_write(*ptep) && !pte_dirty(pte)); | |
| 279 | + if (IS_ENABLED(CONFIG_ARM64_HW_AFDBM) && | |
| 280 | + pte_valid(*ptep) && pte_valid(pte)) { | |
| 281 | + VM_WARN_ONCE(!pte_young(pte), | |
| 282 | + "%s: racy access flag clearing: 0x%016llx -> 0x%016llx", | |
| 283 | + __func__, pte_val(*ptep), pte_val(pte)); | |
| 284 | + VM_WARN_ONCE(pte_write(*ptep) && !pte_dirty(pte), | |
| 285 | + "%s: racy dirty state clearing: 0x%016llx -> 0x%016llx", | |
| 286 | + __func__, pte_val(*ptep), pte_val(pte)); | |
| 283 | 287 | } |
| 284 | 288 | |
| 285 | 289 | set_pte(ptep, pte); |
| @@ -20,6 +20,8 @@ · 19 unmodified lines | ||
| 20 | 20 | #ifndef __ASM_SYSREG_H |
| 21 | 21 | #define __ASM_SYSREG_H |
| 22 | 22 | |
| 23 | +#include <linux/stringify.h> | |
| 24 | + | |
| 23 | 25 | #include <asm/opcodes.h> |
| 24 | 26 | |
| 25 | 27 | /* |
| @@ -208,6 +210,8 @@ · 182 unmodified lines | ||
| 208 | 210 | |
| 209 | 211 | #else |
| 210 | 212 | |
| 213 | +#include <linux/types.h> | |
| 214 | + | |
| 211 | 215 | asm( |
| 212 | 216 | " .irp num,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n" |
| 213 | 217 | " .equ __reg_num_x\\num, \\num\n" |
| @@ -232,6 +236,23 @@ static inline void config_sctlr_el1(u32 clear, u32 set) · 18 unmodified lines | ||
| 232 | 236 | val |= set; |
| 233 | 237 | asm volatile("msr sctlr_el1, %0" : : "r" (val)); |
| 234 | 238 | } |
| 239 | + | |
| 240 | +/* | |
| 241 | + * Unlike read_cpuid, calls to read_sysreg are never expected to be | |
| 242 | + * optimized away or replaced with synthetic values. | |
| 243 | + */ | |
| 244 | +#define read_sysreg(r) ({ \ | |
| 245 | + u64 __val; \ | |
| 246 | + asm volatile("mrs %0, " __stringify(r) : "=r" (__val)); \ | |
| 247 | + __val; \ | |
| 248 | +}) | |
| 249 | + | |
| 250 | +#define write_sysreg(v, r) do { \ | |
| 251 | + u64 __val = (u64)v; \ | |
| 252 | + asm volatile("msr " __stringify(r) ", %0" \ | |
| 253 | + : : "r" (__val)); \ | |
| 254 | +} while (0) | |
| 255 | + | |
| 235 | 256 | #endif |
| 236 | 257 | |
| 237 | 258 | #endif /* __ASM_SYSREG_H */ |
| @@ -108,49 +108,11 @@ int main(void) · 107 unmodified lines | ||
| 108 | 108 | DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs)); |
| 109 | 109 | DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs)); |
| 110 | 110 | DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs)); |
| 111 | − DEFINE(CPU_SP_EL1, offsetof(struct kvm_regs, sp_el1)); | |
| 112 | − DEFINE(CPU_ELR_EL1, offsetof(struct kvm_regs, elr_el1)); | |
| 113 | − DEFINE(CPU_SPSR, offsetof(struct kvm_regs, spsr)); | |
| 114 | − DEFINE(CPU_SYSREGS, offsetof(struct kvm_cpu_context, sys_regs)); | |
| 111 | + DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2])); | |
| 115 | 112 | DEFINE(VCPU_ESR_EL2, offsetof(struct kvm_vcpu, arch.fault.esr_el2)); |
| 116 | 113 | DEFINE(VCPU_FAR_EL2, offsetof(struct kvm_vcpu, arch.fault.far_el2)); |
| 117 | 114 | DEFINE(VCPU_HPFAR_EL2, offsetof(struct kvm_vcpu, arch.fault.hpfar_el2)); |
| 118 | − DEFINE(VCPU_DEBUG_FLAGS, offsetof(struct kvm_vcpu, arch.debug_flags)); | |
| 119 | − DEFINE(VCPU_DEBUG_PTR, offsetof(struct kvm_vcpu, arch.debug_ptr)); | |
| 120 | − DEFINE(DEBUG_BCR, offsetof(struct kvm_guest_debug_arch, dbg_bcr)); | |
| 121 | − DEFINE(DEBUG_BVR, offsetof(struct kvm_guest_debug_arch, dbg_bvr)); | |
| 122 | − DEFINE(DEBUG_WCR, offsetof(struct kvm_guest_debug_arch, dbg_wcr)); | |
| 123 | − DEFINE(DEBUG_WVR, offsetof(struct kvm_guest_debug_arch, dbg_wvr)); | |
| 124 | − DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); | |
| 125 | − DEFINE(VCPU_MDCR_EL2, offsetof(struct kvm_vcpu, arch.mdcr_el2)); | |
| 126 | − DEFINE(VCPU_IRQ_LINES, offsetof(struct kvm_vcpu, arch.irq_lines)); | |
| 127 | 115 | DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context)); |
| 128 | − DEFINE(VCPU_HOST_DEBUG_STATE, offsetof(struct kvm_vcpu, arch.host_debug_state)); | |
| 129 | − DEFINE(VCPU_TIMER_CNTV_CTL, offsetof(struct kvm_vcpu, arch.timer_cpu.cntv_ctl)); | |
| 130 | − DEFINE(VCPU_TIMER_CNTV_CVAL, offsetof(struct kvm_vcpu, arch.timer_cpu.cntv_cval)); | |
| 131 | − DEFINE(KVM_TIMER_CNTVOFF, offsetof(struct kvm, arch.timer.cntvoff)); | |
| 132 | − DEFINE(KVM_TIMER_ENABLED, offsetof(struct kvm, arch.timer.enabled)); | |
| 133 | − DEFINE(VCPU_KVM, offsetof(struct kvm_vcpu, kvm)); | |
| 134 | − DEFINE(VCPU_VGIC_CPU, offsetof(struct kvm_vcpu, arch.vgic_cpu)); | |
| 135 | − DEFINE(VGIC_V2_CPU_HCR, offsetof(struct vgic_cpu, vgic_v2.vgic_hcr)); | |
| 136 | − DEFINE(VGIC_V2_CPU_VMCR, offsetof(struct vgic_cpu, vgic_v2.vgic_vmcr)); | |
| 137 | − DEFINE(VGIC_V2_CPU_MISR, offsetof(struct vgic_cpu, vgic_v2.vgic_misr)); | |
| 138 | − DEFINE(VGIC_V2_CPU_EISR, offsetof(struct vgic_cpu, vgic_v2.vgic_eisr)); | |
| 139 | − DEFINE(VGIC_V2_CPU_ELRSR, offsetof(struct vgic_cpu, vgic_v2.vgic_elrsr)); | |
| 140 | − DEFINE(VGIC_V2_CPU_APR, offsetof(struct vgic_cpu, vgic_v2.vgic_apr)); | |
| 141 | − DEFINE(VGIC_V2_CPU_LR, offsetof(struct vgic_cpu, vgic_v2.vgic_lr)); | |
| 142 | − DEFINE(VGIC_V3_CPU_SRE, offsetof(struct vgic_cpu, vgic_v3.vgic_sre)); | |
| 143 | − DEFINE(VGIC_V3_CPU_HCR, offsetof(struct vgic_cpu, vgic_v3.vgic_hcr)); | |
| 144 | − DEFINE(VGIC_V3_CPU_VMCR, offsetof(struct vgic_cpu, vgic_v3.vgic_vmcr)); | |
| 145 | − DEFINE(VGIC_V3_CPU_MISR, offsetof(struct vgic_cpu, vgic_v3.vgic_misr)); | |
| 146 | − DEFINE(VGIC_V3_CPU_EISR, offsetof(struct vgic_cpu, vgic_v3.vgic_eisr)); | |
| 147 | − DEFINE(VGIC_V3_CPU_ELRSR, offsetof(struct vgic_cpu, vgic_v3.vgic_elrsr)); | |
| 148 | − DEFINE(VGIC_V3_CPU_AP0R, offsetof(struct vgic_cpu, vgic_v3.vgic_ap0r)); | |
| 149 | − DEFINE(VGIC_V3_CPU_AP1R, offsetof(struct vgic_cpu, vgic_v3.vgic_ap1r)); | |
| 150 | − DEFINE(VGIC_V3_CPU_LR, offsetof(struct vgic_cpu, vgic_v3.vgic_lr)); | |
| 151 | − DEFINE(VGIC_CPU_NR_LR, offsetof(struct vgic_cpu, nr_lr)); | |
| 152 | − DEFINE(KVM_VTTBR, offsetof(struct kvm, arch.vttbr)); | |
| 153 | − DEFINE(KVM_VGIC_VCTRL, offsetof(struct kvm, arch.vgic.vctrl_base)); | |
| 154 | 116 | #endif |
| 155 | 117 | #ifdef CONFIG_CPU_PM |
| 156 | 118 | DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx)); |
| @@ -44,8 +44,9 @@ unsigned int compat_elf_hwcap2 __read_mostly; · 43 unmodified lines | ||
| 44 | 44 | |
| 45 | 45 | DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); |
| 46 | 46 | |
| 47 | −#define ARM64_FTR_BITS(STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \ | |
| 47 | +#define __ARM64_FTR_BITS(SIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \ | |
| 48 | 48 | { \ |
| 49 | + .sign = SIGNED, \ | |
| 49 | 50 | .strict = STRICT, \ |
| 50 | 51 | .type = TYPE, \ |
| 51 | 52 | .shift = SHIFT, \ |
| @@ -53,6 +54,14 @@ DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS); · 1 unmodified line | ||
| 53 | 54 | .safe_val = SAFE_VAL, \ |
| 54 | 55 | } |
| 55 | 56 | |
| 57 | +/* Define a feature with signed values */ | |
| 58 | +#define ARM64_FTR_BITS(STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \ | |
| 59 | + __ARM64_FTR_BITS(FTR_SIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) | |
| 60 | + | |
| 61 | +/* Define a feature with unsigned value */ | |
| 62 | +#define U_ARM64_FTR_BITS(STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) \ | |
| 63 | + __ARM64_FTR_BITS(FTR_UNSIGNED, STRICT, TYPE, SHIFT, WIDTH, SAFE_VAL) | |
| 64 | + | |
| 56 | 65 | #define ARM64_FTR_END \ |
| 57 | 66 | { \ |
| 58 | 67 | .width = 0, \ |
| @@ -99,7 +108,7 @@ static struct arm64_ftr_bits ftr_id_aa64mmfr0[] = { · 40 unmodified lines | ||
| 99 | 108 | * Differing PARange is fine as long as all peripherals and memory are mapped |
| 100 | 109 | * within the minimum PARange of all CPUs |
| 101 | 110 | */ |
| 102 | − ARM64_FTR_BITS(FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR0_PARANGE_SHIFT, 4, 0), | |
| 111 | + U_ARM64_FTR_BITS(FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR0_PARANGE_SHIFT, 4, 0), | |
| 103 | 112 | ARM64_FTR_END, |
| 104 | 113 | }; |
| 105 | 114 | |
| @@ -115,18 +124,18 @@ static struct arm64_ftr_bits ftr_id_aa64mmfr1[] = { · 9 unmodified lines | ||
| 115 | 124 | }; |
| 116 | 125 | |
| 117 | 126 | static struct arm64_ftr_bits ftr_ctr[] = { |
| 118 | − ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RAO */ | |
| 127 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 31, 1, 1), /* RAO */ | |
| 119 | 128 | ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 28, 3, 0), |
| 120 | − ARM64_FTR_BITS(FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ | |
| 121 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 20, 4, 0), /* ERG */ | |
| 122 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ | |
| 129 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */ | |
| 130 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 20, 4, 0), /* ERG */ | |
| 131 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */ | |
| 123 | 132 | /* |
| 124 | 133 | * Linux can handle differing I-cache policies. Userspace JITs will |
| 125 | 134 | * make use of *minLine |
| 126 | 135 | */ |
| 127 | − ARM64_FTR_BITS(FTR_NONSTRICT, FTR_EXACT, 14, 2, 0), /* L1Ip */ | |
| 136 | + U_ARM64_FTR_BITS(FTR_NONSTRICT, FTR_EXACT, 14, 2, 0), /* L1Ip */ | |
| 128 | 137 | ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 4, 10, 0), /* RAZ */ |
| 129 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 0, 4, 0), /* IminLine */ | |
| 138 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 0, 4, 0), /* IminLine */ | |
| 130 | 139 | ARM64_FTR_END, |
| 131 | 140 | }; |
| 132 | 141 | |
| @@ -144,12 +153,12 @@ static struct arm64_ftr_bits ftr_id_mmfr0[] = { · 11 unmodified lines | ||
| 144 | 153 | |
| 145 | 154 | static struct arm64_ftr_bits ftr_id_aa64dfr0[] = { |
| 146 | 155 | ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 32, 32, 0), |
| 147 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_CTX_CMPS_SHIFT, 4, 0), | |
| 148 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_WRPS_SHIFT, 4, 0), | |
| 149 | − ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_BRPS_SHIFT, 4, 0), | |
| 150 | − ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_PMUVER_SHIFT, 4, 0), | |
| 151 | − ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_TRACEVER_SHIFT, 4, 0), | |
| 152 | − ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_DEBUGVER_SHIFT, 4, 0x6), | |
| 156 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_CTX_CMPS_SHIFT, 4, 0), | |
| 157 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_WRPS_SHIFT, 4, 0), | |
| 158 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, ID_AA64DFR0_BRPS_SHIFT, 4, 0), | |
| 159 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_PMUVER_SHIFT, 4, 0), | |
| 160 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_TRACEVER_SHIFT, 4, 0), | |
| 161 | + U_ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, ID_AA64DFR0_DEBUGVER_SHIFT, 4, 0x6), | |
| 153 | 162 | ARM64_FTR_END, |
| 154 | 163 | }; |
| 155 | 164 | |
| @@ -127,7 +127,11 @@ static int __init uefi_init(void) · 126 unmodified lines | ||
| 127 | 127 | table_size = sizeof(efi_config_table_64_t) * efi.systab->nr_tables; |
| 128 | 128 | config_tables = early_memremap(efi_to_phys(efi.systab->tables), |
| 129 | 129 | table_size); |
| 130 | − | |
| 130 | + if (config_tables == NULL) { | |
| 131 | + pr_warn("Unable to map EFI config table array.\n"); | |
| 132 | + retval = -ENOMEM; | |
| 133 | + goto out; | |
| 134 | + } | |
| 131 | 135 | retval = efi_config_parse_tables(config_tables, efi.systab->nr_tables, |
| 132 | 136 | sizeof(efi_config_table_64_t), NULL); |
| 133 | 137 | |
| @@ -209,6 +213,14 @@ void __init efi_init(void) · 75 unmodified lines | ||
| 209 | 213 | PAGE_ALIGN(params.mmap_size + (params.mmap & ~PAGE_MASK))); |
| 210 | 214 | memmap.phys_map = params.mmap; |
| 211 | 215 | memmap.map = early_memremap(params.mmap, params.mmap_size); |
| 216 | + if (memmap.map == NULL) { | |
| 217 | + /* | |
| 218 | + * If we are booting via UEFI, the UEFI memory map is the only | |
| 219 | + * description of memory we have, so there is little point in | |
| 220 | + * proceeding if we cannot access it. | |
| 221 | + */ | |
| 222 | + panic("Unable to map EFI memory map.\n"); | |
| 223 | + } | |
| 212 | 224 | memmap.map_end = memmap.map + params.mmap_size; |
| 213 | 225 | memmap.desc_size = params.desc_size; |
| 214 | 226 | memmap.desc_version = params.desc_ver; |
| @@ -227,7 +239,6 @@ static bool __init efi_virtmap_init(void) · 12 unmodified lines | ||
| 227 | 239 | init_new_context(NULL, &efi_mm); |
| 228 | 240 | |
| 229 | 241 | for_each_efi_memory_desc(&memmap, md) { |
| 230 | − u64 paddr, npages, size; | |
| 231 | 242 | pgprot_t prot; |
| 232 | 243 | |
| 233 | 244 | if (!(md->attribute & EFI_MEMORY_RUNTIME)) |
| @@ -235,11 +246,6 @@ static bool __init efi_virtmap_init(void) · 1 unmodified line | ||
| 235 | 246 | if (md->virt_addr == 0) |
| 236 | 247 | return false; |
| 237 | 248 | |
| 238 | − paddr = md->phys_addr; | |
| 239 | − npages = md->num_pages; | |
| 240 | − memrange_efi_to_native(&paddr, &npages); | |
| 241 | − size = npages << PAGE_SHIFT; | |
| 242 | − | |
| 243 | 249 | pr_info(" EFI remap 0x%016llx => %p\n", |
| 244 | 250 | md->phys_addr, (void *)md->virt_addr); |
| 245 | 251 | |
| @@ -256,7 +262,8 @@ static bool __init efi_virtmap_init(void) · 10 unmodified lines | ||
| 256 | 262 | else |
| 257 | 263 | prot = PAGE_KERNEL; |
| 258 | 264 | |
| 259 | − create_pgd_mapping(&efi_mm, paddr, md->virt_addr, size, | |
| 265 | + create_pgd_mapping(&efi_mm, md->phys_addr, md->virt_addr, | |
| 266 | + md->num_pages << EFI_PAGE_SHIFT, | |
| 260 | 267 | __pgprot(pgprot_val(prot) | PTE_NG)); |
| 261 | 268 | } |
| 262 | 269 | return true; |
| @@ -273,12 +280,12 @@ static int __init arm64_enable_runtime_services(void) · 10 unmodified lines | ||
| 273 | 280 | |
| 274 | 281 | if (!efi_enabled(EFI_BOOT)) { |
| 275 | 282 | pr_info("EFI services will not be available.\n"); |
| 276 | − return -1; | |
| 283 | + return 0; | |
| 277 | 284 | } |
| 278 | 285 | |
| 279 | 286 | if (efi_runtime_disabled()) { |
| 280 | 287 | pr_info("EFI runtime services will be disabled.\n"); |
| 281 | − return -1; | |
| 288 | + return 0; | |
| 282 | 289 | } |
| 283 | 290 | |
| 284 | 291 | pr_info("Remapping and enabling EFI services.\n"); |
| @@ -288,7 +295,7 @@ static int __init arm64_enable_runtime_services(void) · 3 unmodified lines | ||
| 288 | 295 | mapsize); |
| 289 | 296 | if (!memmap.map) { |
| 290 | 297 | pr_err("Failed to remap EFI memory map\n"); |
| 291 | − return -1; | |
| 298 | + return -ENOMEM; | |
| 292 | 299 | } |
| 293 | 300 | memmap.map_end = memmap.map + mapsize; |
| 294 | 301 | efi.memmap = &memmap; |
| @@ -297,13 +304,13 @@ static int __init arm64_enable_runtime_services(void) · 2 unmodified lines | ||
| 297 | 304 | sizeof(efi_system_table_t)); |
| 298 | 305 | if (!efi.systab) { |
| 299 | 306 | pr_err("Failed to remap EFI System Table\n"); |
| 300 | − return -1; | |
| 307 | + return -ENOMEM; | |
| 301 | 308 | } |
| 302 | 309 | set_bit(EFI_SYSTEM_TABLES, &efi.flags); |
| 303 | 310 | |
| 304 | 311 | if (!efi_virtmap_init()) { |
| 305 | 312 | pr_err("No UEFI virtual mapping was installed -- runtime services will not be available\n"); |
| 306 | − return -1; | |
| 313 | + return -ENOMEM; | |
| 307 | 314 | } |
| 308 | 315 | |
| 309 | 316 | /* Set up runtime services function pointers */ |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | */ |
| 6 | 6 | |
| 7 | 7 | #include <asm-generic/vmlinux.lds.h> |
| 8 | +#include <asm/cache.h> | |
| 8 | 9 | #include <asm/kernel-pgtable.h> |
| 9 | 10 | #include <asm/thread_info.h> |
| 10 | 11 | #include <asm/memory.h> |
| @@ -140,7 +141,7 @@ SECTIONS · 129 unmodified lines | ||
| 140 | 141 | ARM_EXIT_KEEP(EXIT_DATA) |
| 141 | 142 | } |
| 142 | 143 | |
| 143 | − PERCPU_SECTION(64) | |
| 144 | + PERCPU_SECTION(L1_CACHE_BYTES) | |
| 144 | 145 | |
| 145 | 146 | . = ALIGN(PAGE_SIZE); |
| 146 | 147 | __init_end = .; |
| @@ -158,7 +159,7 @@ SECTIONS · 11 unmodified lines | ||
| 158 | 159 | . = ALIGN(PAGE_SIZE); |
| 159 | 160 | _data = .; |
| 160 | 161 | _sdata = .; |
| 161 | − RW_DATA_SECTION(64, PAGE_SIZE, THREAD_SIZE) | |
| 162 | + RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE) | |
| 162 | 163 | PECOFF_EDATA_PADDING |
| 163 | 164 | _edata = .; |
| 164 | 165 | |
| @@ -10,6 +10,7 @@ KVM=../../../virt/kvm · 9 unmodified lines | ||
| 10 | 10 | ARM=../../../arch/arm/kvm |
| 11 | 11 | |
| 12 | 12 | obj-$(CONFIG_KVM_ARM_HOST) += kvm.o |
| 13 | +obj-$(CONFIG_KVM_ARM_HOST) += hyp/ | |
| 13 | 14 | |
| 14 | 15 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o |
| 15 | 16 | kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o |
| @@ -22,8 +23,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += guest.o debug.o reset.o sys_regs.o sys_regs_generi · 6 unmodified lines | ||
| 22 | 23 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o |
| 23 | 24 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o |
| 24 | 25 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2-emul.o |
| 25 | −kvm-$(CONFIG_KVM_ARM_HOST) += vgic-v2-switch.o | |
| 26 | 26 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v3.o |
| 27 | 27 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v3-emul.o |
| 28 | −kvm-$(CONFIG_KVM_ARM_HOST) += vgic-v3-switch.o | |
| 29 | 28 | kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o |
| @@ -28,13 +28,21 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <asm/cputype.h> |
| 29 | 29 | #include <asm/uaccess.h> |
| 30 | 30 | #include <asm/kvm.h> |
| 31 | −#include <asm/kvm_asm.h> | |
| 32 | 31 | #include <asm/kvm_emulate.h> |
| 33 | 32 | #include <asm/kvm_coproc.h> |
| 34 | 33 | |
| 35 | 34 | #include "trace.h" |
| 36 | 35 | |
| 36 | +#define VM_STAT(x) { #x, offsetof(struct kvm, stat.x), KVM_STAT_VM } | |
| 37 | +#define VCPU_STAT(x) { #x, offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU } | |
| 38 | + | |
| 37 | 39 | struct kvm_stats_debugfs_item debugfs_entries[] = { |
| 40 | + VCPU_STAT(hvc_exit_stat), | |
| 41 | + VCPU_STAT(wfe_exit_stat), | |
| 42 | + VCPU_STAT(wfi_exit_stat), | |
| 43 | + VCPU_STAT(mmio_exit_user), | |
| 44 | + VCPU_STAT(mmio_exit_kernel), | |
| 45 | + VCPU_STAT(exits), | |
| 38 | 46 | { NULL } |
| 39 | 47 | }; |
| 40 | 48 | |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/kvm_host.h> |
| 24 | 24 | |
| 25 | 25 | #include <asm/esr.h> |
| 26 | +#include <asm/kvm_asm.h> | |
| 26 | 27 | #include <asm/kvm_coproc.h> |
| 27 | 28 | #include <asm/kvm_emulate.h> |
| 28 | 29 | #include <asm/kvm_mmu.h> |
| @@ -37,8 +38,9 @@ static int handle_hvc(struct kvm_vcpu *vcpu, struct kvm_run *run) · 8 unmodified lines | ||
| 37 | 38 | { |
| 38 | 39 | int ret; |
| 39 | 40 | |
| 40 | − trace_kvm_hvc_arm64(*vcpu_pc(vcpu), *vcpu_reg(vcpu, 0), | |
| 41 | + trace_kvm_hvc_arm64(*vcpu_pc(vcpu), vcpu_get_reg(vcpu, 0), | |
| 41 | 42 | kvm_vcpu_hvc_get_imm(vcpu)); |
| 43 | + vcpu->stat.hvc_exit_stat++; | |
| 42 | 44 | |
| 43 | 45 | ret = kvm_psci_call(vcpu); |
| 44 | 46 | if (ret < 0) { |
| @@ -71,9 +73,11 @@ static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run) · 26 unmodified lines | ||
| 71 | 73 | { |
| 72 | 74 | if (kvm_vcpu_get_hsr(vcpu) & ESR_ELx_WFx_ISS_WFE) { |
| 73 | 75 | trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true); |
| 76 | + vcpu->stat.wfe_exit_stat++; | |
| 74 | 77 | kvm_vcpu_on_spin(vcpu); |
| 75 | 78 | } else { |
| 76 | 79 | trace_kvm_wfx_arm64(*vcpu_pc(vcpu), false); |
| 80 | + vcpu->stat.wfi_exit_stat++; | |
| 77 | 81 | kvm_vcpu_block(vcpu); |
| 78 | 82 | } |
| 79 | 83 | |
| @@ -94,6 +94,15 @@ __do_hyp_init: · 93 unmodified lines | ||
| 94 | 94 | */ |
| 95 | 95 | mrs x5, ID_AA64MMFR0_EL1 |
| 96 | 96 | bfi x4, x5, #16, #3 |
| 97 | + /* | |
| 98 | + * Read the VMIDBits bits from ID_AA64MMFR1_EL1 and set the VS bit in | |
| 99 | + * VTCR_EL2. | |
| 100 | + */ | |
| 101 | + mrs x5, ID_AA64MMFR1_EL1 | |
| 102 | + ubfx x5, x5, #5, #1 | |
| 103 | + lsl x5, x5, #VTCR_EL2_VS | |
| 104 | + orr x4, x4, x5 | |
| 105 | + | |
| 97 | 106 | msr vtcr_el2, x4 |
| 98 | 107 | |
| 99 | 108 | mrs x4, mair_el1 |
| @@ -17,910 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | |
| 18 | 18 | #include <linux/linkage.h> |
| 19 | 19 | |
| 20 | −#include <asm/alternative.h> | |
| 21 | −#include <asm/asm-offsets.h> | |
| 22 | 20 | #include <asm/assembler.h> |
| 23 | −#include <asm/cpufeature.h> | |
| 24 | −#include <asm/debug-monitors.h> | |
| 25 | −#include <asm/esr.h> | |
| 26 | −#include <asm/fpsimdmacros.h> | |
| 27 | −#include <asm/kvm.h> | |
| 28 | −#include <asm/kvm_arm.h> | |
| 29 | −#include <asm/kvm_asm.h> | |
| 30 | −#include <asm/kvm_mmu.h> | |
| 31 | −#include <asm/memory.h> | |
| 32 | − | |
| 33 | −#define CPU_GP_REG_OFFSET(x) (CPU_GP_REGS + x) | |
| 34 | −#define CPU_XREG_OFFSET(x) CPU_GP_REG_OFFSET(CPU_USER_PT_REGS + 8*x) | |
| 35 | −#define CPU_SPSR_OFFSET(x) CPU_GP_REG_OFFSET(CPU_SPSR + 8*x) | |
| 36 | −#define CPU_SYSREG_OFFSET(x) (CPU_SYSREGS + 8*x) | |
| 37 | − | |
| 38 | − .text | |
| 39 | − .pushsection .hyp.text, "ax" | |
| 40 | − .align PAGE_SHIFT | |
| 41 | − | |
| 42 | −.macro save_common_regs | |
| 43 | − // x2: base address for cpu context | |
| 44 | − // x3: tmp register | |
| 45 | − | |
| 46 | − add x3, x2, #CPU_XREG_OFFSET(19) | |
| 47 | − stp x19, x20, [x3] | |
| 48 | − stp x21, x22, [x3, #16] | |
| 49 | − stp x23, x24, [x3, #32] | |
| 50 | − stp x25, x26, [x3, #48] | |
| 51 | − stp x27, x28, [x3, #64] | |
| 52 | − stp x29, lr, [x3, #80] | |
| 53 | − | |
| 54 | − mrs x19, sp_el0 | |
| 55 | − mrs x20, elr_el2 // pc before entering el2 | |
| 56 | − mrs x21, spsr_el2 // pstate before entering el2 | |
| 57 | − | |
| 58 | − stp x19, x20, [x3, #96] | |
| 59 | − str x21, [x3, #112] | |
| 60 | − | |
| 61 | − mrs x22, sp_el1 | |
| 62 | − mrs x23, elr_el1 | |
| 63 | − mrs x24, spsr_el1 | |
| 64 | − | |
| 65 | − str x22, [x2, #CPU_GP_REG_OFFSET(CPU_SP_EL1)] | |
| 66 | − str x23, [x2, #CPU_GP_REG_OFFSET(CPU_ELR_EL1)] | |
| 67 | − str x24, [x2, #CPU_SPSR_OFFSET(KVM_SPSR_EL1)] | |
| 68 | −.endm | |
| 69 | − | |
| 70 | −.macro restore_common_regs | |
| 71 | − // x2: base address for cpu context | |
| 72 | − // x3: tmp register | |
| 73 | − | |
| 74 | − ldr x22, [x2, #CPU_GP_REG_OFFSET(CPU_SP_EL1)] | |
| 75 | − ldr x23, [x2, #CPU_GP_REG_OFFSET(CPU_ELR_EL1)] | |
| 76 | − ldr x24, [x2, #CPU_SPSR_OFFSET(KVM_SPSR_EL1)] | |
| 77 | − | |
| 78 | − msr sp_el1, x22 | |
| 79 | − msr elr_el1, x23 | |
| 80 | − msr spsr_el1, x24 | |
| 81 | − | |
| 82 | − add x3, x2, #CPU_XREG_OFFSET(31) // SP_EL0 | |
| 83 | − ldp x19, x20, [x3] | |
| 84 | − ldr x21, [x3, #16] | |
| 85 | − | |
| 86 | − msr sp_el0, x19 | |
| 87 | − msr elr_el2, x20 // pc on return from el2 | |
| 88 | − msr spsr_el2, x21 // pstate on return from el2 | |
| 89 | − | |
| 90 | − add x3, x2, #CPU_XREG_OFFSET(19) | |
| 91 | − ldp x19, x20, [x3] | |
| 92 | − ldp x21, x22, [x3, #16] | |
| 93 | − ldp x23, x24, [x3, #32] | |
| 94 | − ldp x25, x26, [x3, #48] | |
| 95 | − ldp x27, x28, [x3, #64] | |
| 96 | − ldp x29, lr, [x3, #80] | |
| 97 | −.endm | |
| 98 | − | |
| 99 | −.macro save_host_regs | |
| 100 | − save_common_regs | |
| 101 | −.endm | |
| 102 | − | |
| 103 | −.macro restore_host_regs | |
| 104 | − restore_common_regs | |
| 105 | −.endm | |
| 106 | − | |
| 107 | −.macro save_fpsimd | |
| 108 | − // x2: cpu context address | |
| 109 | − // x3, x4: tmp regs | |
| 110 | − add x3, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS) | |
| 111 | − fpsimd_save x3, 4 | |
| 112 | −.endm | |
| 113 | − | |
| 114 | −.macro restore_fpsimd | |
| 115 | − // x2: cpu context address | |
| 116 | − // x3, x4: tmp regs | |
| 117 | − add x3, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS) | |
| 118 | − fpsimd_restore x3, 4 | |
| 119 | −.endm | |
| 120 | − | |
| 121 | −.macro save_guest_regs | |
| 122 | − // x0 is the vcpu address | |
| 123 | − // x1 is the return code, do not corrupt! | |
| 124 | − // x2 is the cpu context | |
| 125 | − // x3 is a tmp register | |
| 126 | − // Guest's x0-x3 are on the stack | |
| 127 | − | |
| 128 | − // Compute base to save registers | |
| 129 | − add x3, x2, #CPU_XREG_OFFSET(4) | |
| 130 | − stp x4, x5, [x3] | |
| 131 | − stp x6, x7, [x3, #16] | |
| 132 | − stp x8, x9, [x3, #32] | |
| 133 | − stp x10, x11, [x3, #48] | |
| 134 | − stp x12, x13, [x3, #64] | |
| 135 | − stp x14, x15, [x3, #80] | |
| 136 | − stp x16, x17, [x3, #96] | |
| 137 | − str x18, [x3, #112] | |
| 138 | − | |
| 139 | − pop x6, x7 // x2, x3 | |
| 140 | − pop x4, x5 // x0, x1 | |
| 141 | − | |
| 142 | − add x3, x2, #CPU_XREG_OFFSET(0) | |
| 143 | − stp x4, x5, [x3] | |
| 144 | − stp x6, x7, [x3, #16] | |
| 145 | − | |
| 146 | − save_common_regs | |
| 147 | −.endm | |
| 148 | − | |
| 149 | −.macro restore_guest_regs | |
| 150 | − // x0 is the vcpu address. | |
| 151 | − // x2 is the cpu context | |
| 152 | − // x3 is a tmp register | |
| 153 | − | |
| 154 | − // Prepare x0-x3 for later restore | |
| 155 | − add x3, x2, #CPU_XREG_OFFSET(0) | |
| 156 | − ldp x4, x5, [x3] | |
| 157 | − ldp x6, x7, [x3, #16] | |
| 158 | − push x4, x5 // Push x0-x3 on the stack | |
| 159 | − push x6, x7 | |
| 160 | − | |
| 161 | − // x4-x18 | |
| 162 | − ldp x4, x5, [x3, #32] | |
| 163 | − ldp x6, x7, [x3, #48] | |
| 164 | − ldp x8, x9, [x3, #64] | |
| 165 | − ldp x10, x11, [x3, #80] | |
| 166 | − ldp x12, x13, [x3, #96] | |
| 167 | − ldp x14, x15, [x3, #112] | |
| 168 | − ldp x16, x17, [x3, #128] | |
| 169 | − ldr x18, [x3, #144] | |
| 170 | − | |
| 171 | − // x19-x29, lr, sp*, elr*, spsr* | |
| 172 | − restore_common_regs | |
| 173 | − | |
| 174 | − // Last bits of the 64bit state | |
| 175 | − pop x2, x3 | |
| 176 | − pop x0, x1 | |
| 177 | − | |
| 178 | − // Do not touch any register after this! | |
| 179 | −.endm | |
| 180 | − | |
| 181 | −/* | |
| 182 | − * Macros to perform system register save/restore. | |
| 183 | − * | |
| 184 | − * Ordering here is absolutely critical, and must be kept consistent | |
| 185 | − * in {save,restore}_sysregs, {save,restore}_guest_32bit_state, | |
| 186 | − * and in kvm_asm.h. | |
| 187 | − * | |
| 188 | − * In other words, don't touch any of these unless you know what | |
| 189 | − * you are doing. | |
| 190 | − */ | |
| 191 | −.macro save_sysregs | |
| 192 | − // x2: base address for cpu context | |
| 193 | − // x3: tmp register | |
| 194 | − | |
| 195 | − add x3, x2, #CPU_SYSREG_OFFSET(MPIDR_EL1) | |
| 196 | − | |
| 197 | − mrs x4, vmpidr_el2 | |
| 198 | − mrs x5, csselr_el1 | |
| 199 | − mrs x6, sctlr_el1 | |
| 200 | − mrs x7, actlr_el1 | |
| 201 | − mrs x8, cpacr_el1 | |
| 202 | − mrs x9, ttbr0_el1 | |
| 203 | − mrs x10, ttbr1_el1 | |
| 204 | − mrs x11, tcr_el1 | |
| 205 | − mrs x12, esr_el1 | |
| 206 | − mrs x13, afsr0_el1 | |
| 207 | − mrs x14, afsr1_el1 | |
| 208 | − mrs x15, far_el1 | |
| 209 | − mrs x16, mair_el1 | |
| 210 | − mrs x17, vbar_el1 | |
| 211 | − mrs x18, contextidr_el1 | |
| 212 | − mrs x19, tpidr_el0 | |
| 213 | − mrs x20, tpidrro_el0 | |
| 214 | − mrs x21, tpidr_el1 | |
| 215 | − mrs x22, amair_el1 | |
| 216 | − mrs x23, cntkctl_el1 | |
| 217 | − mrs x24, par_el1 | |
| 218 | − mrs x25, mdscr_el1 | |
| 219 | − | |
| 220 | − stp x4, x5, [x3] | |
| 221 | − stp x6, x7, [x3, #16] | |
| 222 | − stp x8, x9, [x3, #32] | |
| 223 | − stp x10, x11, [x3, #48] | |
| 224 | − stp x12, x13, [x3, #64] | |
| 225 | − stp x14, x15, [x3, #80] | |
| 226 | − stp x16, x17, [x3, #96] | |
| 227 | − stp x18, x19, [x3, #112] | |
| 228 | − stp x20, x21, [x3, #128] | |
| 229 | − stp x22, x23, [x3, #144] | |
| 230 | − stp x24, x25, [x3, #160] | |
| 231 | −.endm | |
| 232 | − | |
| 233 | −.macro save_debug type | |
| 234 | − // x4: pointer to register set | |
| 235 | − // x5: number of registers to skip | |
| 236 | − // x6..x22 trashed | |
| 237 | − | |
| 238 | − adr x22, 1f | |
| 239 | − add x22, x22, x5, lsl #2 | |
| 240 | − br x22 | |
| 241 | −1: | |
| 242 | − mrs x21, \type\()15_el1 | |
| 243 | − mrs x20, \type\()14_el1 | |
| 244 | − mrs x19, \type\()13_el1 | |
| 245 | − mrs x18, \type\()12_el1 | |
| 246 | − mrs x17, \type\()11_el1 | |
| 247 | − mrs x16, \type\()10_el1 | |
| 248 | − mrs x15, \type\()9_el1 | |
| 249 | − mrs x14, \type\()8_el1 | |
| 250 | − mrs x13, \type\()7_el1 | |
| 251 | − mrs x12, \type\()6_el1 | |
| 252 | − mrs x11, \type\()5_el1 | |
| 253 | − mrs x10, \type\()4_el1 | |
| 254 | − mrs x9, \type\()3_el1 | |
| 255 | − mrs x8, \type\()2_el1 | |
| 256 | − mrs x7, \type\()1_el1 | |
| 257 | − mrs x6, \type\()0_el1 | |
| 258 | − | |
| 259 | − adr x22, 1f | |
| 260 | − add x22, x22, x5, lsl #2 | |
| 261 | − br x22 | |
| 262 | −1: | |
| 263 | − str x21, [x4, #(15 * 8)] | |
| 264 | − str x20, [x4, #(14 * 8)] | |
| 265 | − str x19, [x4, #(13 * 8)] | |
| 266 | − str x18, [x4, #(12 * 8)] | |
| 267 | − str x17, [x4, #(11 * 8)] | |
| 268 | − str x16, [x4, #(10 * 8)] | |
| 269 | − str x15, [x4, #(9 * 8)] | |
| 270 | − str x14, [x4, #(8 * 8)] | |
| 271 | − str x13, [x4, #(7 * 8)] | |
| 272 | − str x12, [x4, #(6 * 8)] | |
| 273 | − str x11, [x4, #(5 * 8)] | |
| 274 | − str x10, [x4, #(4 * 8)] | |
| 275 | − str x9, [x4, #(3 * 8)] | |
| 276 | − str x8, [x4, #(2 * 8)] | |
| 277 | − str x7, [x4, #(1 * 8)] | |
| 278 | − str x6, [x4, #(0 * 8)] | |
| 279 | −.endm | |
| 280 | − | |
| 281 | −.macro restore_sysregs | |
| 282 | − // x2: base address for cpu context | |
| 283 | − // x3: tmp register | |
| 284 | − | |
| 285 | − add x3, x2, #CPU_SYSREG_OFFSET(MPIDR_EL1) | |
| 286 | − | |
| 287 | − ldp x4, x5, [x3] | |
| 288 | − ldp x6, x7, [x3, #16] | |
| 289 | − ldp x8, x9, [x3, #32] | |
| 290 | − ldp x10, x11, [x3, #48] | |
| 291 | − ldp x12, x13, [x3, #64] | |
| 292 | − ldp x14, x15, [x3, #80] | |
| 293 | − ldp x16, x17, [x3, #96] | |
| 294 | − ldp x18, x19, [x3, #112] | |
| 295 | − ldp x20, x21, [x3, #128] | |
| 296 | − ldp x22, x23, [x3, #144] | |
| 297 | − ldp x24, x25, [x3, #160] | |
| 298 | − | |
| 299 | − msr vmpidr_el2, x4 | |
| 300 | − msr csselr_el1, x5 | |
| 301 | − msr sctlr_el1, x6 | |
| 302 | − msr actlr_el1, x7 | |
| 303 | − msr cpacr_el1, x8 | |
| 304 | − msr ttbr0_el1, x9 | |
| 305 | − msr ttbr1_el1, x10 | |
| 306 | − msr tcr_el1, x11 | |
| 307 | − msr esr_el1, x12 | |
| 308 | − msr afsr0_el1, x13 | |
| 309 | − msr afsr1_el1, x14 | |
| 310 | − msr far_el1, x15 | |
| 311 | − msr mair_el1, x16 | |
| 312 | − msr vbar_el1, x17 | |
| 313 | − msr contextidr_el1, x18 | |
| 314 | − msr tpidr_el0, x19 | |
| 315 | − msr tpidrro_el0, x20 | |
| 316 | − msr tpidr_el1, x21 | |
| 317 | − msr amair_el1, x22 | |
| 318 | − msr cntkctl_el1, x23 | |
| 319 | − msr par_el1, x24 | |
| 320 | − msr mdscr_el1, x25 | |
| 321 | −.endm | |
| 322 | − | |
| 323 | −.macro restore_debug type | |
| 324 | − // x4: pointer to register set | |
| 325 | − // x5: number of registers to skip | |
| 326 | − // x6..x22 trashed | |
| 327 | − | |
| 328 | − adr x22, 1f | |
| 329 | − add x22, x22, x5, lsl #2 | |
| 330 | − br x22 | |
| 331 | −1: | |
| 332 | − ldr x21, [x4, #(15 * 8)] | |
| 333 | − ldr x20, [x4, #(14 * 8)] | |
| 334 | − ldr x19, [x4, #(13 * 8)] | |
| 335 | − ldr x18, [x4, #(12 * 8)] | |
| 336 | − ldr x17, [x4, #(11 * 8)] | |
| 337 | − ldr x16, [x4, #(10 * 8)] | |
| 338 | − ldr x15, [x4, #(9 * 8)] | |
| 339 | − ldr x14, [x4, #(8 * 8)] | |
| 340 | − ldr x13, [x4, #(7 * 8)] | |
| 341 | − ldr x12, [x4, #(6 * 8)] | |
| 342 | − ldr x11, [x4, #(5 * 8)] | |
| 343 | − ldr x10, [x4, #(4 * 8)] | |
| 344 | − ldr x9, [x4, #(3 * 8)] | |
| 345 | − ldr x8, [x4, #(2 * 8)] | |
| 346 | − ldr x7, [x4, #(1 * 8)] | |
| 347 | − ldr x6, [x4, #(0 * 8)] | |
| 348 | − | |
| 349 | − adr x22, 1f | |
| 350 | − add x22, x22, x5, lsl #2 | |
| 351 | − br x22 | |
| 352 | −1: | |
| 353 | − msr \type\()15_el1, x21 | |
| 354 | − msr \type\()14_el1, x20 | |
| 355 | − msr \type\()13_el1, x19 | |
| 356 | − msr \type\()12_el1, x18 | |
| 357 | − msr \type\()11_el1, x17 | |
| 358 | − msr \type\()10_el1, x16 | |
| 359 | − msr \type\()9_el1, x15 | |
| 360 | − msr \type\()8_el1, x14 | |
| 361 | − msr \type\()7_el1, x13 | |
| 362 | − msr \type\()6_el1, x12 | |
| 363 | − msr \type\()5_el1, x11 | |
| 364 | − msr \type\()4_el1, x10 | |
| 365 | − msr \type\()3_el1, x9 | |
| 366 | − msr \type\()2_el1, x8 | |
| 367 | − msr \type\()1_el1, x7 | |
| 368 | − msr \type\()0_el1, x6 | |
| 369 | −.endm | |
| 370 | − | |
| 371 | −.macro skip_32bit_state tmp, target | |
| 372 | − // Skip 32bit state if not needed | |
| 373 | − mrs \tmp, hcr_el2 | |
| 374 | − tbnz \tmp, #HCR_RW_SHIFT, \target | |
| 375 | −.endm | |
| 376 | − | |
| 377 | −.macro skip_tee_state tmp, target | |
| 378 | − // Skip ThumbEE state if not needed | |
| 379 | − mrs \tmp, id_pfr0_el1 | |
| 380 | − tbz \tmp, #12, \target | |
| 381 | −.endm | |
| 382 | − | |
| 383 | −.macro skip_debug_state tmp, target | |
| 384 | − ldr \tmp, [x0, #VCPU_DEBUG_FLAGS] | |
| 385 | − tbz \tmp, #KVM_ARM64_DEBUG_DIRTY_SHIFT, \target | |
| 386 | −.endm | |
| 387 | − | |
| 388 | −/* | |
| 389 | − * Branch to target if CPTR_EL2.TFP bit is set (VFP/SIMD trapping enabled) | |
| 390 | − */ | |
| 391 | −.macro skip_fpsimd_state tmp, target | |
| 392 | − mrs \tmp, cptr_el2 | |
| 393 | − tbnz \tmp, #CPTR_EL2_TFP_SHIFT, \target | |
| 394 | −.endm | |
| 395 | − | |
| 396 | −.macro compute_debug_state target | |
| 397 | − // Compute debug state: If any of KDE, MDE or KVM_ARM64_DEBUG_DIRTY | |
| 398 | − // is set, we do a full save/restore cycle and disable trapping. | |
| 399 | − add x25, x0, #VCPU_CONTEXT | |
| 400 | − | |
| 401 | − // Check the state of MDSCR_EL1 | |
| 402 | − ldr x25, [x25, #CPU_SYSREG_OFFSET(MDSCR_EL1)] | |
| 403 | − and x26, x25, #DBG_MDSCR_KDE | |
| 404 | − and x25, x25, #DBG_MDSCR_MDE | |
| 405 | − adds xzr, x25, x26 | |
| 406 | − b.eq 9998f // Nothing to see there | |
| 407 | − | |
| 408 | − // If any interesting bits was set, we must set the flag | |
| 409 | − mov x26, #KVM_ARM64_DEBUG_DIRTY | |
| 410 | − str x26, [x0, #VCPU_DEBUG_FLAGS] | |
| 411 | − b 9999f // Don't skip restore | |
| 412 | − | |
| 413 | −9998: | |
| 414 | − // Otherwise load the flags from memory in case we recently | |
| 415 | − // trapped | |
| 416 | − skip_debug_state x25, \target | |
| 417 | −9999: | |
| 418 | −.endm | |
| 419 | − | |
| 420 | −.macro save_guest_32bit_state | |
| 421 | − skip_32bit_state x3, 1f | |
| 422 | − | |
| 423 | − add x3, x2, #CPU_SPSR_OFFSET(KVM_SPSR_ABT) | |
| 424 | − mrs x4, spsr_abt | |
| 425 | − mrs x5, spsr_und | |
| 426 | − mrs x6, spsr_irq | |
| 427 | − mrs x7, spsr_fiq | |
| 428 | − stp x4, x5, [x3] | |
| 429 | − stp x6, x7, [x3, #16] | |
| 430 | − | |
| 431 | − add x3, x2, #CPU_SYSREG_OFFSET(DACR32_EL2) | |
| 432 | − mrs x4, dacr32_el2 | |
| 433 | − mrs x5, ifsr32_el2 | |
| 434 | − stp x4, x5, [x3] | |
| 435 | − | |
| 436 | − skip_fpsimd_state x8, 2f | |
| 437 | − mrs x6, fpexc32_el2 | |
| 438 | − str x6, [x3, #16] | |
| 439 | −2: | |
| 440 | − skip_debug_state x8, 1f | |
| 441 | − mrs x7, dbgvcr32_el2 | |
| 442 | − str x7, [x3, #24] | |
| 443 | −1: | |
| 444 | −.endm | |
| 445 | − | |
| 446 | −.macro restore_guest_32bit_state | |
| 447 | − skip_32bit_state x3, 1f | |
| 448 | − | |
| 449 | − add x3, x2, #CPU_SPSR_OFFSET(KVM_SPSR_ABT) | |
| 450 | − ldp x4, x5, [x3] | |
| 451 | − ldp x6, x7, [x3, #16] | |
| 452 | − msr spsr_abt, x4 | |
| 453 | − msr spsr_und, x5 | |
| 454 | − msr spsr_irq, x6 | |
| 455 | − msr spsr_fiq, x7 | |
| 456 | − | |
| 457 | − add x3, x2, #CPU_SYSREG_OFFSET(DACR32_EL2) | |
| 458 | − ldp x4, x5, [x3] | |
| 459 | − msr dacr32_el2, x4 | |
| 460 | − msr ifsr32_el2, x5 | |
| 461 | − | |
| 462 | − skip_debug_state x8, 1f | |
| 463 | − ldr x7, [x3, #24] | |
| 464 | − msr dbgvcr32_el2, x7 | |
| 465 | −1: | |
| 466 | −.endm | |
| 467 | − | |
| 468 | −.macro activate_traps | |
| 469 | − ldr x2, [x0, #VCPU_HCR_EL2] | |
| 470 | − | |
| 471 | − /* | |
| 472 | − * We are about to set CPTR_EL2.TFP to trap all floating point | |
| 473 | − * register accesses to EL2, however, the ARM ARM clearly states that | |
| 474 | − * traps are only taken to EL2 if the operation would not otherwise | |
| 475 | − * trap to EL1. Therefore, always make sure that for 32-bit guests, | |
| 476 | − * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit. | |
| 477 | − */ | |
| 478 | − tbnz x2, #HCR_RW_SHIFT, 99f // open code skip_32bit_state | |
| 479 | − mov x3, #(1 << 30) | |
| 480 | − msr fpexc32_el2, x3 | |
| 481 | − isb | |
| 482 | −99: | |
| 483 | − msr hcr_el2, x2 | |
| 484 | − mov x2, #CPTR_EL2_TTA | |
| 485 | − orr x2, x2, #CPTR_EL2_TFP | |
| 486 | − msr cptr_el2, x2 | |
| 487 | − | |
| 488 | − mov x2, #(1 << 15) // Trap CP15 Cr=15 | |
| 489 | − msr hstr_el2, x2 | |
| 490 | − | |
| 491 | − // Monitor Debug Config - see kvm_arm_setup_debug() | |
| 492 | − ldr x2, [x0, #VCPU_MDCR_EL2] | |
| 493 | − msr mdcr_el2, x2 | |
| 494 | −.endm | |
| 495 | − | |
| 496 | −.macro deactivate_traps | |
| 497 | − mov x2, #HCR_RW | |
| 498 | − msr hcr_el2, x2 | |
| 499 | − msr hstr_el2, xzr | |
| 500 | − | |
| 501 | − mrs x2, mdcr_el2 | |
| 502 | − and x2, x2, #MDCR_EL2_HPMN_MASK | |
| 503 | − msr mdcr_el2, x2 | |
| 504 | −.endm | |
| 505 | − | |
| 506 | −.macro activate_vm | |
| 507 | − ldr x1, [x0, #VCPU_KVM] | |
| 508 | − kern_hyp_va x1 | |
| 509 | − ldr x2, [x1, #KVM_VTTBR] | |
| 510 | − msr vttbr_el2, x2 | |
| 511 | −.endm | |
| 512 | − | |
| 513 | −.macro deactivate_vm | |
| 514 | − msr vttbr_el2, xzr | |
| 515 | −.endm | |
| 516 | − | |
| 517 | −/* | |
| 518 | − * Call into the vgic backend for state saving | |
| 519 | − */ | |
| 520 | −.macro save_vgic_state | |
| 521 | −alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF | |
| 522 | − bl __save_vgic_v2_state | |
| 523 | −alternative_else | |
| 524 | − bl __save_vgic_v3_state | |
| 525 | −alternative_endif | |
| 526 | − mrs x24, hcr_el2 | |
| 527 | − mov x25, #HCR_INT_OVERRIDE | |
| 528 | − neg x25, x25 | |
| 529 | − and x24, x24, x25 | |
| 530 | − msr hcr_el2, x24 | |
| 531 | −.endm | |
| 532 | − | |
| 533 | −/* | |
| 534 | − * Call into the vgic backend for state restoring | |
| 535 | − */ | |
| 536 | −.macro restore_vgic_state | |
| 537 | − mrs x24, hcr_el2 | |
| 538 | − ldr x25, [x0, #VCPU_IRQ_LINES] | |
| 539 | − orr x24, x24, #HCR_INT_OVERRIDE | |
| 540 | − orr x24, x24, x25 | |
| 541 | − msr hcr_el2, x24 | |
| 542 | −alternative_if_not ARM64_HAS_SYSREG_GIC_CPUIF | |
| 543 | − bl __restore_vgic_v2_state | |
| 544 | −alternative_else | |
| 545 | − bl __restore_vgic_v3_state | |
| 546 | −alternative_endif | |
| 547 | −.endm | |
| 548 | − | |
| 549 | −.macro save_timer_state | |
| 550 | − // x0: vcpu pointer | |
| 551 | − ldr x2, [x0, #VCPU_KVM] | |
| 552 | − kern_hyp_va x2 | |
| 553 | − ldr w3, [x2, #KVM_TIMER_ENABLED] | |
| 554 | − cbz w3, 1f | |
| 555 | − | |
| 556 | − mrs x3, cntv_ctl_el0 | |
| 557 | − and x3, x3, #3 | |
| 558 | − str w3, [x0, #VCPU_TIMER_CNTV_CTL] | |
| 559 | − | |
| 560 | − isb | |
| 561 | − | |
| 562 | − mrs x3, cntv_cval_el0 | |
| 563 | − str x3, [x0, #VCPU_TIMER_CNTV_CVAL] | |
| 564 | − | |
| 565 | −1: | |
| 566 | − // Disable the virtual timer | |
| 567 | − msr cntv_ctl_el0, xzr | |
| 568 | − | |
| 569 | − // Allow physical timer/counter access for the host | |
| 570 | − mrs x2, cnthctl_el2 | |
| 571 | − orr x2, x2, #3 | |
| 572 | − msr cnthctl_el2, x2 | |
| 573 | − | |
| 574 | − // Clear cntvoff for the host | |
| 575 | − msr cntvoff_el2, xzr | |
| 576 | −.endm | |
| 577 | − | |
| 578 | −.macro restore_timer_state | |
| 579 | − // x0: vcpu pointer | |
| 580 | − // Disallow physical timer access for the guest | |
| 581 | − // Physical counter access is allowed | |
| 582 | − mrs x2, cnthctl_el2 | |
| 583 | − orr x2, x2, #1 | |
| 584 | − bic x2, x2, #2 | |
| 585 | − msr cnthctl_el2, x2 | |
| 586 | − | |
| 587 | − ldr x2, [x0, #VCPU_KVM] | |
| 588 | − kern_hyp_va x2 | |
| 589 | − ldr w3, [x2, #KVM_TIMER_ENABLED] | |
| 590 | − cbz w3, 1f | |
| 591 | − | |
| 592 | − ldr x3, [x2, #KVM_TIMER_CNTVOFF] | |
| 593 | − msr cntvoff_el2, x3 | |
| 594 | − ldr x2, [x0, #VCPU_TIMER_CNTV_CVAL] | |
| 595 | − msr cntv_cval_el0, x2 | |
| 596 | − isb | |
| 597 | − | |
| 598 | − ldr w2, [x0, #VCPU_TIMER_CNTV_CTL] | |
| 599 | − and x2, x2, #3 | |
| 600 | − msr cntv_ctl_el0, x2 | |
| 601 | −1: | |
| 602 | −.endm | |
| 603 | − | |
| 604 | −__save_sysregs: | |
| 605 | − save_sysregs | |
| 606 | − ret | |
| 607 | − | |
| 608 | −__restore_sysregs: | |
| 609 | − restore_sysregs | |
| 610 | − ret | |
| 611 | − | |
| 612 | −/* Save debug state */ | |
| 613 | −__save_debug: | |
| 614 | − // x2: ptr to CPU context | |
| 615 | − // x3: ptr to debug reg struct | |
| 616 | − // x4/x5/x6-22/x24-26: trashed | |
| 617 | − | |
| 618 | − mrs x26, id_aa64dfr0_el1 | |
| 619 | − ubfx x24, x26, #12, #4 // Extract BRPs | |
| 620 | − ubfx x25, x26, #20, #4 // Extract WRPs | |
| 621 | − mov w26, #15 | |
| 622 | − sub w24, w26, w24 // How many BPs to skip | |
| 623 | − sub w25, w26, w25 // How many WPs to skip | |
| 624 | − | |
| 625 | − mov x5, x24 | |
| 626 | − add x4, x3, #DEBUG_BCR | |
| 627 | − save_debug dbgbcr | |
| 628 | − add x4, x3, #DEBUG_BVR | |
| 629 | − save_debug dbgbvr | |
| 630 | − | |
| 631 | − mov x5, x25 | |
| 632 | − add x4, x3, #DEBUG_WCR | |
| 633 | − save_debug dbgwcr | |
| 634 | − add x4, x3, #DEBUG_WVR | |
| 635 | − save_debug dbgwvr | |
| 636 | − | |
| 637 | − mrs x21, mdccint_el1 | |
| 638 | − str x21, [x2, #CPU_SYSREG_OFFSET(MDCCINT_EL1)] | |
| 639 | − ret | |
| 640 | − | |
| 641 | −/* Restore debug state */ | |
| 642 | −__restore_debug: | |
| 643 | − // x2: ptr to CPU context | |
| 644 | − // x3: ptr to debug reg struct | |
| 645 | − // x4/x5/x6-22/x24-26: trashed | |
| 646 | − | |
| 647 | − mrs x26, id_aa64dfr0_el1 | |
| 648 | − ubfx x24, x26, #12, #4 // Extract BRPs | |
| 649 | − ubfx x25, x26, #20, #4 // Extract WRPs | |
| 650 | − mov w26, #15 | |
| 651 | − sub w24, w26, w24 // How many BPs to skip | |
| 652 | − sub w25, w26, w25 // How many WPs to skip | |
| 653 | − | |
| 654 | − mov x5, x24 | |
| 655 | − add x4, x3, #DEBUG_BCR | |
| 656 | − restore_debug dbgbcr | |
| 657 | − add x4, x3, #DEBUG_BVR | |
| 658 | − restore_debug dbgbvr | |
| 659 | − | |
| 660 | − mov x5, x25 | |
| 661 | − add x4, x3, #DEBUG_WCR | |
| 662 | − restore_debug dbgwcr | |
| 663 | − add x4, x3, #DEBUG_WVR | |
| 664 | − restore_debug dbgwvr | |
| 665 | − | |
| 666 | − ldr x21, [x2, #CPU_SYSREG_OFFSET(MDCCINT_EL1)] | |
| 667 | − msr mdccint_el1, x21 | |
| 668 | − | |
| 669 | − ret | |
| 670 | − | |
| 671 | −__save_fpsimd: | |
| 672 | − skip_fpsimd_state x3, 1f | |
| 673 | − save_fpsimd | |
| 674 | −1: ret | |
| 675 | − | |
| 676 | −__restore_fpsimd: | |
| 677 | − skip_fpsimd_state x3, 1f | |
| 678 | − restore_fpsimd | |
| 679 | −1: ret | |
| 680 | − | |
| 681 | −switch_to_guest_fpsimd: | |
| 682 | − push x4, lr | |
| 683 | − | |
| 684 | − mrs x2, cptr_el2 | |
| 685 | − bic x2, x2, #CPTR_EL2_TFP | |
| 686 | − msr cptr_el2, x2 | |
| 687 | − isb | |
| 688 | − | |
| 689 | − mrs x0, tpidr_el2 | |
| 690 | − | |
| 691 | − ldr x2, [x0, #VCPU_HOST_CONTEXT] | |
| 692 | − kern_hyp_va x2 | |
| 693 | − bl __save_fpsimd | |
| 694 | − | |
| 695 | − add x2, x0, #VCPU_CONTEXT | |
| 696 | − bl __restore_fpsimd | |
| 697 | − | |
| 698 | − skip_32bit_state x3, 1f | |
| 699 | − ldr x4, [x2, #CPU_SYSREG_OFFSET(FPEXC32_EL2)] | |
| 700 | − msr fpexc32_el2, x4 | |
| 701 | −1: | |
| 702 | − pop x4, lr | |
| 703 | − pop x2, x3 | |
| 704 | − pop x0, x1 | |
| 705 | − | |
| 706 | − eret | |
| 707 | − | |
| 708 | −/* | |
| 709 | − * u64 __kvm_vcpu_run(struct kvm_vcpu *vcpu); | |
| 710 | − * | |
| 711 | − * This is the world switch. The first half of the function | |
| 712 | − * deals with entering the guest, and anything from __kvm_vcpu_return | |
| 713 | − * to the end of the function deals with reentering the host. | |
| 714 | − * On the enter path, only x0 (vcpu pointer) must be preserved until | |
| 715 | − * the last moment. On the exit path, x0 (vcpu pointer) and x1 (exception | |
| 716 | − * code) must both be preserved until the epilogue. | |
| 717 | − * In both cases, x2 points to the CPU context we're saving/restoring from/to. | |
| 718 | − */ | |
| 719 | −ENTRY(__kvm_vcpu_run) | |
| 720 | − kern_hyp_va x0 | |
| 721 | − msr tpidr_el2, x0 // Save the vcpu register | |
| 722 | − | |
| 723 | − // Host context | |
| 724 | − ldr x2, [x0, #VCPU_HOST_CONTEXT] | |
| 725 | − kern_hyp_va x2 | |
| 726 | − | |
| 727 | − save_host_regs | |
| 728 | − bl __save_sysregs | |
| 729 | − | |
| 730 | − compute_debug_state 1f | |
| 731 | − add x3, x0, #VCPU_HOST_DEBUG_STATE | |
| 732 | − bl __save_debug | |
| 733 | −1: | |
| 734 | − activate_traps | |
| 735 | − activate_vm | |
| 736 | − | |
| 737 | − restore_vgic_state | |
| 738 | − restore_timer_state | |
| 739 | − | |
| 740 | − // Guest context | |
| 741 | − add x2, x0, #VCPU_CONTEXT | |
| 742 | − | |
| 743 | − // We must restore the 32-bit state before the sysregs, thanks | |
| 744 | − // to Cortex-A57 erratum #852523. | |
| 745 | − restore_guest_32bit_state | |
| 746 | − bl __restore_sysregs | |
| 747 | − | |
| 748 | − skip_debug_state x3, 1f | |
| 749 | − ldr x3, [x0, #VCPU_DEBUG_PTR] | |
| 750 | − kern_hyp_va x3 | |
| 751 | − bl __restore_debug | |
| 752 | −1: | |
| 753 | − restore_guest_regs | |
| 754 | − | |
| 755 | − // That's it, no more messing around. | |
| 756 | − eret | |
| 757 | − | |
| 758 | −__kvm_vcpu_return: | |
| 759 | − // Assume x0 is the vcpu pointer, x1 the return code | |
| 760 | − // Guest's x0-x3 are on the stack | |
| 761 | − | |
| 762 | − // Guest context | |
| 763 | − add x2, x0, #VCPU_CONTEXT | |
| 764 | − | |
| 765 | − save_guest_regs | |
| 766 | − bl __save_fpsimd | |
| 767 | − bl __save_sysregs | |
| 768 | − | |
| 769 | − skip_debug_state x3, 1f | |
| 770 | − ldr x3, [x0, #VCPU_DEBUG_PTR] | |
| 771 | − kern_hyp_va x3 | |
| 772 | − bl __save_debug | |
| 773 | −1: | |
| 774 | − save_guest_32bit_state | |
| 775 | − | |
| 776 | − save_timer_state | |
| 777 | − save_vgic_state | |
| 778 | − | |
| 779 | − deactivate_traps | |
| 780 | − deactivate_vm | |
| 781 | − | |
| 782 | − // Host context | |
| 783 | − ldr x2, [x0, #VCPU_HOST_CONTEXT] | |
| 784 | − kern_hyp_va x2 | |
| 785 | − | |
| 786 | − bl __restore_sysregs | |
| 787 | − bl __restore_fpsimd | |
| 788 | − /* Clear FPSIMD and Trace trapping */ | |
| 789 | − msr cptr_el2, xzr | |
| 790 | − | |
| 791 | − skip_debug_state x3, 1f | |
| 792 | − // Clear the dirty flag for the next run, as all the state has | |
| 793 | − // already been saved. Note that we nuke the whole 64bit word. | |
| 794 | − // If we ever add more flags, we'll have to be more careful... | |
| 795 | − str xzr, [x0, #VCPU_DEBUG_FLAGS] | |
| 796 | − add x3, x0, #VCPU_HOST_DEBUG_STATE | |
| 797 | − bl __restore_debug | |
| 798 | −1: | |
| 799 | − restore_host_regs | |
| 800 | − | |
| 801 | − mov x0, x1 | |
| 802 | − ret | |
| 803 | −END(__kvm_vcpu_run) | |
| 804 | − | |
| 805 | −// void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); | |
| 806 | −ENTRY(__kvm_tlb_flush_vmid_ipa) | |
| 807 | − dsb ishst | |
| 808 | − | |
| 809 | − kern_hyp_va x0 | |
| 810 | − ldr x2, [x0, #KVM_VTTBR] | |
| 811 | − msr vttbr_el2, x2 | |
| 812 | − isb | |
| 813 | − | |
| 814 | − /* | |
| 815 | − * We could do so much better if we had the VA as well. | |
| 816 | − * Instead, we invalidate Stage-2 for this IPA, and the | |
| 817 | − * whole of Stage-1. Weep... | |
| 818 | − */ | |
| 819 | − lsr x1, x1, #12 | |
| 820 | − tlbi ipas2e1is, x1 | |
| 821 | − /* | |
| 822 | − * We have to ensure completion of the invalidation at Stage-2, | |
| 823 | − * since a table walk on another CPU could refill a TLB with a | |
| 824 | − * complete (S1 + S2) walk based on the old Stage-2 mapping if | |
| 825 | − * the Stage-1 invalidation happened first. | |
| 826 | − */ | |
| 827 | − dsb ish | |
| 828 | − tlbi vmalle1is | |
| 829 | − dsb ish | |
| 830 | − isb | |
| 831 | − | |
| 832 | − msr vttbr_el2, xzr | |
| 833 | − ret | |
| 834 | −ENDPROC(__kvm_tlb_flush_vmid_ipa) | |
| 835 | − | |
| 836 | −/** | |
| 837 | − * void __kvm_tlb_flush_vmid(struct kvm *kvm) - Flush per-VMID TLBs | |
| 838 | − * @struct kvm *kvm - pointer to kvm structure | |
| 839 | − * | |
| 840 | − * Invalidates all Stage 1 and 2 TLB entries for current VMID. | |
| 841 | − */ | |
| 842 | −ENTRY(__kvm_tlb_flush_vmid) | |
| 843 | − dsb ishst | |
| 844 | − | |
| 845 | − kern_hyp_va x0 | |
| 846 | − ldr x2, [x0, #KVM_VTTBR] | |
| 847 | − msr vttbr_el2, x2 | |
| 848 | − isb | |
| 849 | − | |
| 850 | − tlbi vmalls12e1is | |
| 851 | − dsb ish | |
| 852 | − isb | |
| 853 | − | |
| 854 | − msr vttbr_el2, xzr | |
| 855 | − ret | |
| 856 | −ENDPROC(__kvm_tlb_flush_vmid) | |
| 857 | − | |
| 858 | −ENTRY(__kvm_flush_vm_context) | |
| 859 | − dsb ishst | |
| 860 | − tlbi alle1is | |
| 861 | − ic ialluis | |
| 862 | − dsb ish | |
| 863 | − ret | |
| 864 | −ENDPROC(__kvm_flush_vm_context) | |
| 865 | − | |
| 866 | −__kvm_hyp_panic: | |
| 867 | − // Stash PAR_EL1 before corrupting it in __restore_sysregs | |
| 868 | − mrs x0, par_el1 | |
| 869 | − push x0, xzr | |
| 870 | − | |
| 871 | − // Guess the context by looking at VTTBR: | |
| 872 | − // If zero, then we're already a host. | |
| 873 | − // Otherwise restore a minimal host context before panicing. | |
| 874 | − mrs x0, vttbr_el2 | |
| 875 | − cbz x0, 1f | |
| 876 | − | |
| 877 | − mrs x0, tpidr_el2 | |
| 878 | − | |
| 879 | − deactivate_traps | |
| 880 | − deactivate_vm | |
| 881 | − | |
| 882 | − ldr x2, [x0, #VCPU_HOST_CONTEXT] | |
| 883 | − kern_hyp_va x2 | |
| 884 | − | |
| 885 | − bl __restore_sysregs | |
| 886 | − | |
| 887 | − /* | |
| 888 | − * Make sure we have a valid host stack, and don't leave junk in the | |
| 889 | − * frame pointer that will give us a misleading host stack unwinding. | |
| 890 | − */ | |
| 891 | − ldr x22, [x2, #CPU_GP_REG_OFFSET(CPU_SP_EL1)] | |
| 892 | − msr sp_el1, x22 | |
| 893 | − mov x29, xzr | |
| 894 | − | |
| 895 | −1: adr x0, __hyp_panic_str | |
| 896 | − adr x1, 2f | |
| 897 | − ldp x2, x3, [x1] | |
| 898 | − sub x0, x0, x2 | |
| 899 | − add x0, x0, x3 | |
| 900 | − mrs x1, spsr_el2 | |
| 901 | − mrs x2, elr_el2 | |
| 902 | − mrs x3, esr_el2 | |
| 903 | − mrs x4, far_el2 | |
| 904 | − mrs x5, hpfar_el2 | |
| 905 | − pop x6, xzr // active context PAR_EL1 | |
| 906 | − mrs x7, tpidr_el2 | |
| 907 | − | |
| 908 | − mov lr, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT |\ | |
| 909 | − PSR_MODE_EL1h) | |
| 910 | − msr spsr_el2, lr | |
| 911 | − ldr lr, =panic | |
| 912 | − msr elr_el2, lr | |
| 913 | − eret | |
| 914 | − | |
| 915 | − .align 3 | |
| 916 | −2: .quad HYP_PAGE_OFFSET | |
| 917 | − .quad PAGE_OFFSET | |
| 918 | −ENDPROC(__kvm_hyp_panic) | |
| 919 | − | |
| 920 | −__hyp_panic_str: | |
| 921 | − .ascii "HYP panic:\nPS:%08x PC:%016x ESR:%08x\nFAR:%016x HPFAR:%016x PAR:%016x\nVCPU:%p\n\0" | |
| 922 | − | |
| 923 | − .align 2 | |
| 924 | 21 | |
| 925 | 22 | /* |
| 926 | 23 | * u64 kvm_call_hyp(void *hypfn, ...); |
| @@ -934,7 +31,7 @@ __hyp_panic_str: · 7 unmodified lines | ||
| 934 | 31 | * passed as x0, x1, and x2 (a maximum of 3 arguments in addition to the |
| 935 | 32 | * function pointer can be passed). The function being called must be mapped |
| 936 | 33 | * in Hyp mode (see init_hyp_mode in arch/arm/kvm/arm.c). Return values are |
| 937 | − * passed in r0 and r1. | |
| 34 | + * passed in x0. | |
| 938 | 35 | * |
| 939 | 36 | * A function pointer with a value of 0 has a special meaning, and is |
| 940 | 37 | * used to implement __hyp_get_vectors in the same way as in |
| @@ -944,179 +41,3 @@ ENTRY(kvm_call_hyp) · 3 unmodified lines | ||
| 944 | 41 | hvc #0 |
| 945 | 42 | ret |
| 946 | 43 | ENDPROC(kvm_call_hyp) |
| 947 | − | |
| 948 | −.macro invalid_vector label, target | |
| 949 | − .align 2 | |
| 950 | −\label: | |
| 951 | − b \target | |
| 952 | −ENDPROC(\label) | |
| 953 | −.endm | |
| 954 | − | |
| 955 | − /* None of these should ever happen */ | |
| 956 | − invalid_vector el2t_sync_invalid, __kvm_hyp_panic | |
| 957 | − invalid_vector el2t_irq_invalid, __kvm_hyp_panic | |
| 958 | − invalid_vector el2t_fiq_invalid, __kvm_hyp_panic | |
| 959 | − invalid_vector el2t_error_invalid, __kvm_hyp_panic | |
| 960 | − invalid_vector el2h_sync_invalid, __kvm_hyp_panic | |
| 961 | − invalid_vector el2h_irq_invalid, __kvm_hyp_panic | |
| 962 | − invalid_vector el2h_fiq_invalid, __kvm_hyp_panic | |
| 963 | − invalid_vector el2h_error_invalid, __kvm_hyp_panic | |
| 964 | − invalid_vector el1_sync_invalid, __kvm_hyp_panic | |
| 965 | − invalid_vector el1_irq_invalid, __kvm_hyp_panic | |
| 966 | − invalid_vector el1_fiq_invalid, __kvm_hyp_panic | |
| 967 | − invalid_vector el1_error_invalid, __kvm_hyp_panic | |
| 968 | − | |
| 969 | −el1_sync: // Guest trapped into EL2 | |
| 970 | − push x0, x1 | |
| 971 | − push x2, x3 | |
| 972 | − | |
| 973 | − mrs x1, esr_el2 | |
| 974 | − lsr x2, x1, #ESR_ELx_EC_SHIFT | |
| 975 | − | |
| 976 | − cmp x2, #ESR_ELx_EC_HVC64 | |
| 977 | − b.ne el1_trap | |
| 978 | − | |
| 979 | − mrs x3, vttbr_el2 // If vttbr is valid, the 64bit guest | |
| 980 | − cbnz x3, el1_trap // called HVC | |
| 981 | − | |
| 982 | − /* Here, we're pretty sure the host called HVC. */ | |
| 983 | − pop x2, x3 | |
| 984 | − pop x0, x1 | |
| 985 | − | |
| 986 | − /* Check for __hyp_get_vectors */ | |
| 987 | − cbnz x0, 1f | |
| 988 | − mrs x0, vbar_el2 | |
| 989 | − b 2f | |
| 990 | − | |
| 991 | −1: push lr, xzr | |
| 992 | − | |
| 993 | − /* | |
| 994 | − * Compute the function address in EL2, and shuffle the parameters. | |
| 995 | − */ | |
| 996 | − kern_hyp_va x0 | |
| 997 | − mov lr, x0 | |
| 998 | − mov x0, x1 | |
| 999 | − mov x1, x2 | |
| 1000 | − mov x2, x3 | |
| 1001 | − blr lr | |
| 1002 | − | |
| 1003 | − pop lr, xzr | |
| 1004 | −2: eret | |
| 1005 | − | |
| 1006 | −el1_trap: | |
| 1007 | − /* | |
| 1008 | − * x1: ESR | |
| 1009 | − * x2: ESR_EC | |
| 1010 | − */ | |
| 1011 | − | |
| 1012 | − /* Guest accessed VFP/SIMD registers, save host, restore Guest */ | |
| 1013 | − cmp x2, #ESR_ELx_EC_FP_ASIMD | |
| 1014 | − b.eq switch_to_guest_fpsimd | |
| 1015 | − | |
| 1016 | − cmp x2, #ESR_ELx_EC_DABT_LOW | |
| 1017 | − mov x0, #ESR_ELx_EC_IABT_LOW | |
| 1018 | − ccmp x2, x0, #4, ne | |
| 1019 | − b.ne 1f // Not an abort we care about | |
| 1020 | − | |
| 1021 | − /* This is an abort. Check for permission fault */ | |
| 1022 | −alternative_if_not ARM64_WORKAROUND_834220 | |
| 1023 | − and x2, x1, #ESR_ELx_FSC_TYPE | |
| 1024 | − cmp x2, #FSC_PERM | |
| 1025 | − b.ne 1f // Not a permission fault | |
| 1026 | −alternative_else | |
| 1027 | − nop // Use the permission fault path to | |
| 1028 | − nop // check for a valid S1 translation, | |
| 1029 | − nop // regardless of the ESR value. | |
| 1030 | −alternative_endif | |
| 1031 | − | |
| 1032 | − /* | |
| 1033 | − * Check for Stage-1 page table walk, which is guaranteed | |
| 1034 | − * to give a valid HPFAR_EL2. | |
| 1035 | − */ | |
| 1036 | − tbnz x1, #7, 1f // S1PTW is set | |
| 1037 | − | |
| 1038 | − /* Preserve PAR_EL1 */ | |
| 1039 | − mrs x3, par_el1 | |
| 1040 | − push x3, xzr | |
| 1041 | − | |
| 1042 | − /* | |
| 1043 | − * Permission fault, HPFAR_EL2 is invalid. | |
| 1044 | − * Resolve the IPA the hard way using the guest VA. | |
| 1045 | − * Stage-1 translation already validated the memory access rights. | |
| 1046 | − * As such, we can use the EL1 translation regime, and don't have | |
| 1047 | − * to distinguish between EL0 and EL1 access. | |
| 1048 | − */ | |
| 1049 | − mrs x2, far_el2 | |
| 1050 | − at s1e1r, x2 | |
| 1051 | − isb | |
| 1052 | − | |
| 1053 | − /* Read result */ | |
| 1054 | − mrs x3, par_el1 | |
| 1055 | − pop x0, xzr // Restore PAR_EL1 from the stack | |
| 1056 | − msr par_el1, x0 | |
| 1057 | − tbnz x3, #0, 3f // Bail out if we failed the translation | |
| 1058 | − ubfx x3, x3, #12, #36 // Extract IPA | |
| 1059 | − lsl x3, x3, #4 // and present it like HPFAR | |
| 1060 | − b 2f | |
| 1061 | − | |
| 1062 | −1: mrs x3, hpfar_el2 | |
| 1063 | − mrs x2, far_el2 | |
| 1064 | − | |
| 1065 | −2: mrs x0, tpidr_el2 | |
| 1066 | − str w1, [x0, #VCPU_ESR_EL2] | |
| 1067 | − str x2, [x0, #VCPU_FAR_EL2] | |
| 1068 | − str x3, [x0, #VCPU_HPFAR_EL2] | |
| 1069 | − | |
| 1070 | − mov x1, #ARM_EXCEPTION_TRAP | |
| 1071 | − b __kvm_vcpu_return | |
| 1072 | − | |
| 1073 | − /* | |
| 1074 | − * Translation failed. Just return to the guest and | |
| 1075 | − * let it fault again. Another CPU is probably playing | |
| 1076 | − * behind our back. | |
| 1077 | − */ | |
| 1078 | −3: pop x2, x3 | |
| 1079 | − pop x0, x1 | |
| 1080 | − | |
| 1081 | − eret | |
| 1082 | − | |
| 1083 | −el1_irq: | |
| 1084 | − push x0, x1 | |
| 1085 | − push x2, x3 | |
| 1086 | − mrs x0, tpidr_el2 | |
| 1087 | − mov x1, #ARM_EXCEPTION_IRQ | |
| 1088 | − b __kvm_vcpu_return | |
| 1089 | − | |
| 1090 | − .ltorg | |
| 1091 | − | |
| 1092 | − .align 11 | |
| 1093 | − | |
| 1094 | −ENTRY(__kvm_hyp_vector) | |
| 1095 | − ventry el2t_sync_invalid // Synchronous EL2t | |
| 1096 | − ventry el2t_irq_invalid // IRQ EL2t | |
| 1097 | − ventry el2t_fiq_invalid // FIQ EL2t | |
| 1098 | − ventry el2t_error_invalid // Error EL2t | |
| 1099 | − | |
| 1100 | − ventry el2h_sync_invalid // Synchronous EL2h | |
| 1101 | − ventry el2h_irq_invalid // IRQ EL2h | |
| 1102 | − ventry el2h_fiq_invalid // FIQ EL2h | |
| 1103 | − ventry el2h_error_invalid // Error EL2h | |
| 1104 | − | |
| 1105 | − ventry el1_sync // Synchronous 64-bit EL1 | |
| 1106 | − ventry el1_irq // IRQ 64-bit EL1 | |
| 1107 | − ventry el1_fiq_invalid // FIQ 64-bit EL1 | |
| 1108 | − ventry el1_error_invalid // Error 64-bit EL1 | |
| 1109 | − | |
| 1110 | − ventry el1_sync // Synchronous 32-bit EL1 | |
| 1111 | − ventry el1_irq // IRQ 32-bit EL1 | |
| 1112 | − ventry el1_fiq_invalid // FIQ 32-bit EL1 | |
| 1113 | − ventry el1_error_invalid // Error 32-bit EL1 | |
| 1114 | −ENDPROC(__kvm_hyp_vector) | |
| 1115 | − | |
| 1116 | − | |
| 1117 | −ENTRY(__kvm_get_mdcr_el2) | |
| 1118 | − mrs x0, mdcr_el2 | |
| 1119 | − ret | |
| 1120 | −ENDPROC(__kvm_get_mdcr_el2) | |
| 1121 | − | |
| 1122 | − .popsection | |
| @@ -0,0 +1,14 @@ | ||
| 1 | +# | |
| 2 | +# Makefile for Kernel-based Virtual Machine module, HYP part | |
| 3 | +# | |
| 4 | + | |
| 5 | +obj-$(CONFIG_KVM_ARM_HOST) += vgic-v2-sr.o | |
| 6 | +obj-$(CONFIG_KVM_ARM_HOST) += vgic-v3-sr.o | |
| 7 | +obj-$(CONFIG_KVM_ARM_HOST) += timer-sr.o | |
| 8 | +obj-$(CONFIG_KVM_ARM_HOST) += sysreg-sr.o | |
| 9 | +obj-$(CONFIG_KVM_ARM_HOST) += debug-sr.o | |
| 10 | +obj-$(CONFIG_KVM_ARM_HOST) += entry.o | |
| 11 | +obj-$(CONFIG_KVM_ARM_HOST) += switch.o | |
| 12 | +obj-$(CONFIG_KVM_ARM_HOST) += fpsimd.o | |
| 13 | +obj-$(CONFIG_KVM_ARM_HOST) += tlb.o | |
| 14 | +obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o | |
| @@ -0,0 +1,140 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/compiler.h> | |
| 19 | +#include <linux/kvm_host.h> | |
| 20 | + | |
| 21 | +#include <asm/kvm_asm.h> | |
| 22 | +#include <asm/kvm_mmu.h> | |
| 23 | + | |
| 24 | +#include "hyp.h" | |
| 25 | + | |
| 26 | +#define read_debug(r,n) read_sysreg(r##n##_el1) | |
| 27 | +#define write_debug(v,r,n) write_sysreg(v, r##n##_el1) | |
| 28 | + | |
| 29 | +#define save_debug(ptr,reg,nr) \ | |
| 30 | + switch (nr) { \ | |
| 31 | + case 15: ptr[15] = read_debug(reg, 15); \ | |
| 32 | + case 14: ptr[14] = read_debug(reg, 14); \ | |
| 33 | + case 13: ptr[13] = read_debug(reg, 13); \ | |
| 34 | + case 12: ptr[12] = read_debug(reg, 12); \ | |
| 35 | + case 11: ptr[11] = read_debug(reg, 11); \ | |
| 36 | + case 10: ptr[10] = read_debug(reg, 10); \ | |
| 37 | + case 9: ptr[9] = read_debug(reg, 9); \ | |
| 38 | + case 8: ptr[8] = read_debug(reg, 8); \ | |
| 39 | + case 7: ptr[7] = read_debug(reg, 7); \ | |
| 40 | + case 6: ptr[6] = read_debug(reg, 6); \ | |
| 41 | + case 5: ptr[5] = read_debug(reg, 5); \ | |
| 42 | + case 4: ptr[4] = read_debug(reg, 4); \ | |
| 43 | + case 3: ptr[3] = read_debug(reg, 3); \ | |
| 44 | + case 2: ptr[2] = read_debug(reg, 2); \ | |
| 45 | + case 1: ptr[1] = read_debug(reg, 1); \ | |
| 46 | + default: ptr[0] = read_debug(reg, 0); \ | |
| 47 | + } | |
| 48 | + | |
| 49 | +#define restore_debug(ptr,reg,nr) \ | |
| 50 | + switch (nr) { \ | |
| 51 | + case 15: write_debug(ptr[15], reg, 15); \ | |
| 52 | + case 14: write_debug(ptr[14], reg, 14); \ | |
| 53 | + case 13: write_debug(ptr[13], reg, 13); \ | |
| 54 | + case 12: write_debug(ptr[12], reg, 12); \ | |
| 55 | + case 11: write_debug(ptr[11], reg, 11); \ | |
| 56 | + case 10: write_debug(ptr[10], reg, 10); \ | |
| 57 | + case 9: write_debug(ptr[9], reg, 9); \ | |
| 58 | + case 8: write_debug(ptr[8], reg, 8); \ | |
| 59 | + case 7: write_debug(ptr[7], reg, 7); \ | |
| 60 | + case 6: write_debug(ptr[6], reg, 6); \ | |
| 61 | + case 5: write_debug(ptr[5], reg, 5); \ | |
| 62 | + case 4: write_debug(ptr[4], reg, 4); \ | |
| 63 | + case 3: write_debug(ptr[3], reg, 3); \ | |
| 64 | + case 2: write_debug(ptr[2], reg, 2); \ | |
| 65 | + case 1: write_debug(ptr[1], reg, 1); \ | |
| 66 | + default: write_debug(ptr[0], reg, 0); \ | |
| 67 | + } | |
| 68 | + | |
| 69 | +void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu, | |
| 70 | + struct kvm_guest_debug_arch *dbg, | |
| 71 | + struct kvm_cpu_context *ctxt) | |
| 72 | +{ | |
| 73 | + u64 aa64dfr0; | |
| 74 | + int brps, wrps; | |
| 75 | + | |
| 76 | + if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)) | |
| 77 | + return; | |
| 78 | + | |
| 79 | + aa64dfr0 = read_sysreg(id_aa64dfr0_el1); | |
| 80 | + brps = (aa64dfr0 >> 12) & 0xf; | |
| 81 | + wrps = (aa64dfr0 >> 20) & 0xf; | |
| 82 | + | |
| 83 | + save_debug(dbg->dbg_bcr, dbgbcr, brps); | |
| 84 | + save_debug(dbg->dbg_bvr, dbgbvr, brps); | |
| 85 | + save_debug(dbg->dbg_wcr, dbgwcr, wrps); | |
| 86 | + save_debug(dbg->dbg_wvr, dbgwvr, wrps); | |
| 87 | + | |
| 88 | + ctxt->sys_regs[MDCCINT_EL1] = read_sysreg(mdccint_el1); | |
| 89 | +} | |
| 90 | + | |
| 91 | +void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu, | |
| 92 | + struct kvm_guest_debug_arch *dbg, | |
| 93 | + struct kvm_cpu_context *ctxt) | |
| 94 | +{ | |
| 95 | + u64 aa64dfr0; | |
| 96 | + int brps, wrps; | |
| 97 | + | |
| 98 | + if (!(vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)) | |
| 99 | + return; | |
| 100 | + | |
| 101 | + aa64dfr0 = read_sysreg(id_aa64dfr0_el1); | |
| 102 | + | |
| 103 | + brps = (aa64dfr0 >> 12) & 0xf; | |
| 104 | + wrps = (aa64dfr0 >> 20) & 0xf; | |
| 105 | + | |
| 106 | + restore_debug(dbg->dbg_bcr, dbgbcr, brps); | |
| 107 | + restore_debug(dbg->dbg_bvr, dbgbvr, brps); | |
| 108 | + restore_debug(dbg->dbg_wcr, dbgwcr, wrps); | |
| 109 | + restore_debug(dbg->dbg_wvr, dbgwvr, wrps); | |
| 110 | + | |
| 111 | + write_sysreg(ctxt->sys_regs[MDCCINT_EL1], mdccint_el1); | |
| 112 | +} | |
| 113 | + | |
| 114 | +void __hyp_text __debug_cond_save_host_state(struct kvm_vcpu *vcpu) | |
| 115 | +{ | |
| 116 | + /* If any of KDE, MDE or KVM_ARM64_DEBUG_DIRTY is set, perform | |
| 117 | + * a full save/restore cycle. */ | |
| 118 | + if ((vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_KDE) || | |
| 119 | + (vcpu->arch.ctxt.sys_regs[MDSCR_EL1] & DBG_MDSCR_MDE)) | |
| 120 | + vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; | |
| 121 | + | |
| 122 | + __debug_save_state(vcpu, &vcpu->arch.host_debug_state, | |
| 123 | + kern_hyp_va(vcpu->arch.host_cpu_context)); | |
| 124 | +} | |
| 125 | + | |
| 126 | +void __hyp_text __debug_cond_restore_host_state(struct kvm_vcpu *vcpu) | |
| 127 | +{ | |
| 128 | + __debug_restore_state(vcpu, &vcpu->arch.host_debug_state, | |
| 129 | + kern_hyp_va(vcpu->arch.host_cpu_context)); | |
| 130 | + | |
| 131 | + if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY) | |
| 132 | + vcpu->arch.debug_flags &= ~KVM_ARM64_DEBUG_DIRTY; | |
| 133 | +} | |
| 134 | + | |
| 135 | +static u32 __hyp_text __debug_read_mdcr_el2(void) | |
| 136 | +{ | |
| 137 | + return read_sysreg(mdcr_el2); | |
| 138 | +} | |
| 139 | + | |
| 140 | +__alias(__debug_read_mdcr_el2) u32 __kvm_get_mdcr_el2(void); | |
| @@ -0,0 +1,160 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/linkage.h> | |
| 19 | + | |
| 20 | +#include <asm/asm-offsets.h> | |
| 21 | +#include <asm/assembler.h> | |
| 22 | +#include <asm/fpsimdmacros.h> | |
| 23 | +#include <asm/kvm.h> | |
| 24 | +#include <asm/kvm_arm.h> | |
| 25 | +#include <asm/kvm_asm.h> | |
| 26 | +#include <asm/kvm_mmu.h> | |
| 27 | + | |
| 28 | +#define CPU_GP_REG_OFFSET(x) (CPU_GP_REGS + x) | |
| 29 | +#define CPU_XREG_OFFSET(x) CPU_GP_REG_OFFSET(CPU_USER_PT_REGS + 8*x) | |
| 30 | + | |
| 31 | + .text | |
| 32 | + .pushsection .hyp.text, "ax" | |
| 33 | + | |
| 34 | +.macro save_callee_saved_regs ctxt | |
| 35 | + stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] | |
| 36 | + stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] | |
| 37 | + stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] | |
| 38 | + stp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] | |
| 39 | + stp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] | |
| 40 | + stp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] | |
| 41 | +.endm | |
| 42 | + | |
| 43 | +.macro restore_callee_saved_regs ctxt | |
| 44 | + ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] | |
| 45 | + ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] | |
| 46 | + ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)] | |
| 47 | + ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] | |
| 48 | + ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] | |
| 49 | + ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] | |
| 50 | +.endm | |
| 51 | + | |
| 52 | +/* | |
| 53 | + * u64 __guest_enter(struct kvm_vcpu *vcpu, | |
| 54 | + * struct kvm_cpu_context *host_ctxt); | |
| 55 | + */ | |
| 56 | +ENTRY(__guest_enter) | |
| 57 | + // x0: vcpu | |
| 58 | + // x1: host/guest context | |
| 59 | + // x2-x18: clobbered by macros | |
| 60 | + | |
| 61 | + // Store the host regs | |
| 62 | + save_callee_saved_regs x1 | |
| 63 | + | |
| 64 | + // Preserve vcpu & host_ctxt for use at exit time | |
| 65 | + stp x0, x1, [sp, #-16]! | |
| 66 | + | |
| 67 | + add x1, x0, #VCPU_CONTEXT | |
| 68 | + | |
| 69 | + // Prepare x0-x1 for later restore by pushing them onto the stack | |
| 70 | + ldp x2, x3, [x1, #CPU_XREG_OFFSET(0)] | |
| 71 | + stp x2, x3, [sp, #-16]! | |
| 72 | + | |
| 73 | + // x2-x18 | |
| 74 | + ldp x2, x3, [x1, #CPU_XREG_OFFSET(2)] | |
| 75 | + ldp x4, x5, [x1, #CPU_XREG_OFFSET(4)] | |
| 76 | + ldp x6, x7, [x1, #CPU_XREG_OFFSET(6)] | |
| 77 | + ldp x8, x9, [x1, #CPU_XREG_OFFSET(8)] | |
| 78 | + ldp x10, x11, [x1, #CPU_XREG_OFFSET(10)] | |
| 79 | + ldp x12, x13, [x1, #CPU_XREG_OFFSET(12)] | |
| 80 | + ldp x14, x15, [x1, #CPU_XREG_OFFSET(14)] | |
| 81 | + ldp x16, x17, [x1, #CPU_XREG_OFFSET(16)] | |
| 82 | + ldr x18, [x1, #CPU_XREG_OFFSET(18)] | |
| 83 | + | |
| 84 | + // x19-x29, lr | |
| 85 | + restore_callee_saved_regs x1 | |
| 86 | + | |
| 87 | + // Last bits of the 64bit state | |
| 88 | + ldp x0, x1, [sp], #16 | |
| 89 | + | |
| 90 | + // Do not touch any register after this! | |
| 91 | + eret | |
| 92 | +ENDPROC(__guest_enter) | |
| 93 | + | |
| 94 | +ENTRY(__guest_exit) | |
| 95 | + // x0: vcpu | |
| 96 | + // x1: return code | |
| 97 | + // x2-x3: free | |
| 98 | + // x4-x29,lr: vcpu regs | |
| 99 | + // vcpu x0-x3 on the stack | |
| 100 | + | |
| 101 | + add x2, x0, #VCPU_CONTEXT | |
| 102 | + | |
| 103 | + stp x4, x5, [x2, #CPU_XREG_OFFSET(4)] | |
| 104 | + stp x6, x7, [x2, #CPU_XREG_OFFSET(6)] | |
| 105 | + stp x8, x9, [x2, #CPU_XREG_OFFSET(8)] | |
| 106 | + stp x10, x11, [x2, #CPU_XREG_OFFSET(10)] | |
| 107 | + stp x12, x13, [x2, #CPU_XREG_OFFSET(12)] | |
| 108 | + stp x14, x15, [x2, #CPU_XREG_OFFSET(14)] | |
| 109 | + stp x16, x17, [x2, #CPU_XREG_OFFSET(16)] | |
| 110 | + str x18, [x2, #CPU_XREG_OFFSET(18)] | |
| 111 | + | |
| 112 | + ldp x6, x7, [sp], #16 // x2, x3 | |
| 113 | + ldp x4, x5, [sp], #16 // x0, x1 | |
| 114 | + | |
| 115 | + stp x4, x5, [x2, #CPU_XREG_OFFSET(0)] | |
| 116 | + stp x6, x7, [x2, #CPU_XREG_OFFSET(2)] | |
| 117 | + | |
| 118 | + save_callee_saved_regs x2 | |
| 119 | + | |
| 120 | + // Restore vcpu & host_ctxt from the stack | |
| 121 | + // (preserving return code in x1) | |
| 122 | + ldp x0, x2, [sp], #16 | |
| 123 | + // Now restore the host regs | |
| 124 | + restore_callee_saved_regs x2 | |
| 125 | + | |
| 126 | + mov x0, x1 | |
| 127 | + ret | |
| 128 | +ENDPROC(__guest_exit) | |
| 129 | + | |
| 130 | +ENTRY(__fpsimd_guest_restore) | |
| 131 | + stp x4, lr, [sp, #-16]! | |
| 132 | + | |
| 133 | + mrs x2, cptr_el2 | |
| 134 | + bic x2, x2, #CPTR_EL2_TFP | |
| 135 | + msr cptr_el2, x2 | |
| 136 | + isb | |
| 137 | + | |
| 138 | + mrs x3, tpidr_el2 | |
| 139 | + | |
| 140 | + ldr x0, [x3, #VCPU_HOST_CONTEXT] | |
| 141 | + kern_hyp_va x0 | |
| 142 | + add x0, x0, #CPU_GP_REG_OFFSET(CPU_FP_REGS) | |
| 143 | + bl __fpsimd_save_state | |
| 144 | + | |
| 145 | + add x2, x3, #VCPU_CONTEXT | |
| 146 | + add x0, x2, #CPU_GP_REG_OFFSET(CPU_FP_REGS) | |
| 147 | + bl __fpsimd_restore_state | |
| 148 | + | |
| 149 | + // Skip restoring fpexc32 for AArch64 guests | |
| 150 | + mrs x1, hcr_el2 | |
| 151 | + tbnz x1, #HCR_RW_SHIFT, 1f | |
| 152 | + ldr x4, [x3, #VCPU_FPEXC32_EL2] | |
| 153 | + msr fpexc32_el2, x4 | |
| 154 | +1: | |
| 155 | + ldp x4, lr, [sp], #16 | |
| 156 | + ldp x2, x3, [sp], #16 | |
| 157 | + ldp x0, x1, [sp], #16 | |
| 158 | + | |
| 159 | + eret | |
| 160 | +ENDPROC(__fpsimd_guest_restore) | |
| @@ -0,0 +1,33 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/linkage.h> | |
| 19 | + | |
| 20 | +#include <asm/fpsimdmacros.h> | |
| 21 | + | |
| 22 | + .text | |
| 23 | + .pushsection .hyp.text, "ax" | |
| 24 | + | |
| 25 | +ENTRY(__fpsimd_save_state) | |
| 26 | + fpsimd_save x0, 1 | |
| 27 | + ret | |
| 28 | +ENDPROC(__fpsimd_save_state) | |
| 29 | + | |
| 30 | +ENTRY(__fpsimd_restore_state) | |
| 31 | + fpsimd_restore x0, 1 | |
| 32 | + ret | |
| 33 | +ENDPROC(__fpsimd_restore_state) | |
| @@ -0,0 +1,212 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/linkage.h> | |
| 19 | + | |
| 20 | +#include <asm/alternative.h> | |
| 21 | +#include <asm/assembler.h> | |
| 22 | +#include <asm/asm-offsets.h> | |
| 23 | +#include <asm/cpufeature.h> | |
| 24 | +#include <asm/kvm_arm.h> | |
| 25 | +#include <asm/kvm_asm.h> | |
| 26 | +#include <asm/kvm_mmu.h> | |
| 27 | + | |
| 28 | + .text | |
| 29 | + .pushsection .hyp.text, "ax" | |
| 30 | + | |
| 31 | +.macro save_x0_to_x3 | |
| 32 | + stp x0, x1, [sp, #-16]! | |
| 33 | + stp x2, x3, [sp, #-16]! | |
| 34 | +.endm | |
| 35 | + | |
| 36 | +.macro restore_x0_to_x3 | |
| 37 | + ldp x2, x3, [sp], #16 | |
| 38 | + ldp x0, x1, [sp], #16 | |
| 39 | +.endm | |
| 40 | + | |
| 41 | +el1_sync: // Guest trapped into EL2 | |
| 42 | + save_x0_to_x3 | |
| 43 | + | |
| 44 | + mrs x1, esr_el2 | |
| 45 | + lsr x2, x1, #ESR_ELx_EC_SHIFT | |
| 46 | + | |
| 47 | + cmp x2, #ESR_ELx_EC_HVC64 | |
| 48 | + b.ne el1_trap | |
| 49 | + | |
| 50 | + mrs x3, vttbr_el2 // If vttbr is valid, the 64bit guest | |
| 51 | + cbnz x3, el1_trap // called HVC | |
| 52 | + | |
| 53 | + /* Here, we're pretty sure the host called HVC. */ | |
| 54 | + restore_x0_to_x3 | |
| 55 | + | |
| 56 | + /* Check for __hyp_get_vectors */ | |
| 57 | + cbnz x0, 1f | |
| 58 | + mrs x0, vbar_el2 | |
| 59 | + b 2f | |
| 60 | + | |
| 61 | +1: stp lr, xzr, [sp, #-16]! | |
| 62 | + | |
| 63 | + /* | |
| 64 | + * Compute the function address in EL2, and shuffle the parameters. | |
| 65 | + */ | |
| 66 | + kern_hyp_va x0 | |
| 67 | + mov lr, x0 | |
| 68 | + mov x0, x1 | |
| 69 | + mov x1, x2 | |
| 70 | + mov x2, x3 | |
| 71 | + blr lr | |
| 72 | + | |
| 73 | + ldp lr, xzr, [sp], #16 | |
| 74 | +2: eret | |
| 75 | + | |
| 76 | +el1_trap: | |
| 77 | + /* | |
| 78 | + * x1: ESR | |
| 79 | + * x2: ESR_EC | |
| 80 | + */ | |
| 81 | + | |
| 82 | + /* Guest accessed VFP/SIMD registers, save host, restore Guest */ | |
| 83 | + cmp x2, #ESR_ELx_EC_FP_ASIMD | |
| 84 | + b.eq __fpsimd_guest_restore | |
| 85 | + | |
| 86 | + cmp x2, #ESR_ELx_EC_DABT_LOW | |
| 87 | + mov x0, #ESR_ELx_EC_IABT_LOW | |
| 88 | + ccmp x2, x0, #4, ne | |
| 89 | + b.ne 1f // Not an abort we care about | |
| 90 | + | |
| 91 | + /* This is an abort. Check for permission fault */ | |
| 92 | +alternative_if_not ARM64_WORKAROUND_834220 | |
| 93 | + and x2, x1, #ESR_ELx_FSC_TYPE | |
| 94 | + cmp x2, #FSC_PERM | |
| 95 | + b.ne 1f // Not a permission fault | |
| 96 | +alternative_else | |
| 97 | + nop // Use the permission fault path to | |
| 98 | + nop // check for a valid S1 translation, | |
| 99 | + nop // regardless of the ESR value. | |
| 100 | +alternative_endif | |
| 101 | + | |
| 102 | + /* | |
| 103 | + * Check for Stage-1 page table walk, which is guaranteed | |
| 104 | + * to give a valid HPFAR_EL2. | |
| 105 | + */ | |
| 106 | + tbnz x1, #7, 1f // S1PTW is set | |
| 107 | + | |
| 108 | + /* Preserve PAR_EL1 */ | |
| 109 | + mrs x3, par_el1 | |
| 110 | + stp x3, xzr, [sp, #-16]! | |
| 111 | + | |
| 112 | + /* | |
| 113 | + * Permission fault, HPFAR_EL2 is invalid. | |
| 114 | + * Resolve the IPA the hard way using the guest VA. | |
| 115 | + * Stage-1 translation already validated the memory access rights. | |
| 116 | + * As such, we can use the EL1 translation regime, and don't have | |
| 117 | + * to distinguish between EL0 and EL1 access. | |
| 118 | + */ | |
| 119 | + mrs x2, far_el2 | |
| 120 | + at s1e1r, x2 | |
| 121 | + isb | |
| 122 | + | |
| 123 | + /* Read result */ | |
| 124 | + mrs x3, par_el1 | |
| 125 | + ldp x0, xzr, [sp], #16 // Restore PAR_EL1 from the stack | |
| 126 | + msr par_el1, x0 | |
| 127 | + tbnz x3, #0, 3f // Bail out if we failed the translation | |
| 128 | + ubfx x3, x3, #12, #36 // Extract IPA | |
| 129 | + lsl x3, x3, #4 // and present it like HPFAR | |
| 130 | + b 2f | |
| 131 | + | |
| 132 | +1: mrs x3, hpfar_el2 | |
| 133 | + mrs x2, far_el2 | |
| 134 | + | |
| 135 | +2: mrs x0, tpidr_el2 | |
| 136 | + str w1, [x0, #VCPU_ESR_EL2] | |
| 137 | + str x2, [x0, #VCPU_FAR_EL2] | |
| 138 | + str x3, [x0, #VCPU_HPFAR_EL2] | |
| 139 | + | |
| 140 | + mov x1, #ARM_EXCEPTION_TRAP | |
| 141 | + b __guest_exit | |
| 142 | + | |
| 143 | + /* | |
| 144 | + * Translation failed. Just return to the guest and | |
| 145 | + * let it fault again. Another CPU is probably playing | |
| 146 | + * behind our back. | |
| 147 | + */ | |
| 148 | +3: restore_x0_to_x3 | |
| 149 | + | |
| 150 | + eret | |
| 151 | + | |
| 152 | +el1_irq: | |
| 153 | + save_x0_to_x3 | |
| 154 | + mrs x0, tpidr_el2 | |
| 155 | + mov x1, #ARM_EXCEPTION_IRQ | |
| 156 | + b __guest_exit | |
| 157 | + | |
| 158 | +ENTRY(__hyp_do_panic) | |
| 159 | + mov lr, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT |\ | |
| 160 | + PSR_MODE_EL1h) | |
| 161 | + msr spsr_el2, lr | |
| 162 | + ldr lr, =panic | |
| 163 | + msr elr_el2, lr | |
| 164 | + eret | |
| 165 | +ENDPROC(__hyp_do_panic) | |
| 166 | + | |
| 167 | +.macro invalid_vector label, target = __hyp_panic | |
| 168 | + .align 2 | |
| 169 | +\label: | |
| 170 | + b \target | |
| 171 | +ENDPROC(\label) | |
| 172 | +.endm | |
| 173 | + | |
| 174 | + /* None of these should ever happen */ | |
| 175 | + invalid_vector el2t_sync_invalid | |
| 176 | + invalid_vector el2t_irq_invalid | |
| 177 | + invalid_vector el2t_fiq_invalid | |
| 178 | + invalid_vector el2t_error_invalid | |
| 179 | + invalid_vector el2h_sync_invalid | |
| 180 | + invalid_vector el2h_irq_invalid | |
| 181 | + invalid_vector el2h_fiq_invalid | |
| 182 | + invalid_vector el2h_error_invalid | |
| 183 | + invalid_vector el1_sync_invalid | |
| 184 | + invalid_vector el1_irq_invalid | |
| 185 | + invalid_vector el1_fiq_invalid | |
| 186 | + invalid_vector el1_error_invalid | |
| 187 | + | |
| 188 | + .ltorg | |
| 189 | + | |
| 190 | + .align 11 | |
| 191 | + | |
| 192 | +ENTRY(__kvm_hyp_vector) | |
| 193 | + ventry el2t_sync_invalid // Synchronous EL2t | |
| 194 | + ventry el2t_irq_invalid // IRQ EL2t | |
| 195 | + ventry el2t_fiq_invalid // FIQ EL2t | |
| 196 | + ventry el2t_error_invalid // Error EL2t | |
| 197 | + | |
| 198 | + ventry el2h_sync_invalid // Synchronous EL2h | |
| 199 | + ventry el2h_irq_invalid // IRQ EL2h | |
| 200 | + ventry el2h_fiq_invalid // FIQ EL2h | |
| 201 | + ventry el2h_error_invalid // Error EL2h | |
| 202 | + | |
| 203 | + ventry el1_sync // Synchronous 64-bit EL1 | |
| 204 | + ventry el1_irq // IRQ 64-bit EL1 | |
| 205 | + ventry el1_fiq_invalid // FIQ 64-bit EL1 | |
| 206 | + ventry el1_error_invalid // Error 64-bit EL1 | |
| 207 | + | |
| 208 | + ventry el1_sync // Synchronous 32-bit EL1 | |
| 209 | + ventry el1_irq // IRQ 32-bit EL1 | |
| 210 | + ventry el1_fiq_invalid // FIQ 32-bit EL1 | |
| 211 | + ventry el1_error_invalid // Error 32-bit EL1 | |
| 212 | +ENDPROC(__kvm_hyp_vector) | |
| @@ -0,0 +1,90 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#ifndef __ARM64_KVM_HYP_H__ | |
| 19 | +#define __ARM64_KVM_HYP_H__ | |
| 20 | + | |
| 21 | +#include <linux/compiler.h> | |
| 22 | +#include <linux/kvm_host.h> | |
| 23 | +#include <asm/kvm_mmu.h> | |
| 24 | +#include <asm/sysreg.h> | |
| 25 | + | |
| 26 | +#define __hyp_text __section(.hyp.text) notrace | |
| 27 | + | |
| 28 | +#define kern_hyp_va(v) (typeof(v))((unsigned long)(v) & HYP_PAGE_OFFSET_MASK) | |
| 29 | +#define hyp_kern_va(v) (typeof(v))((unsigned long)(v) - HYP_PAGE_OFFSET \ | |
| 30 | + + PAGE_OFFSET) | |
| 31 | + | |
| 32 | +/** | |
| 33 | + * hyp_alternate_select - Generates patchable code sequences that are | |
| 34 | + * used to switch between two implementations of a function, depending | |
| 35 | + * on the availability of a feature. | |
| 36 | + * | |
| 37 | + * @fname: a symbol name that will be defined as a function returning a | |
| 38 | + * function pointer whose type will match @orig and @alt | |
| 39 | + * @orig: A pointer to the default function, as returned by @fname when | |
| 40 | + * @cond doesn't hold | |
| 41 | + * @alt: A pointer to the alternate function, as returned by @fname | |
| 42 | + * when @cond holds | |
| 43 | + * @cond: a CPU feature (as described in asm/cpufeature.h) | |
| 44 | + */ | |
| 45 | +#define hyp_alternate_select(fname, orig, alt, cond) \ | |
| 46 | +typeof(orig) * __hyp_text fname(void) \ | |
| 47 | +{ \ | |
| 48 | + typeof(alt) *val = orig; \ | |
| 49 | + asm volatile(ALTERNATIVE("nop \n", \ | |
| 50 | + "mov %0, %1 \n", \ | |
| 51 | + cond) \ | |
| 52 | + : "+r" (val) : "r" (alt)); \ | |
| 53 | + return val; \ | |
| 54 | +} | |
| 55 | + | |
| 56 | +void __vgic_v2_save_state(struct kvm_vcpu *vcpu); | |
| 57 | +void __vgic_v2_restore_state(struct kvm_vcpu *vcpu); | |
| 58 | + | |
| 59 | +void __vgic_v3_save_state(struct kvm_vcpu *vcpu); | |
| 60 | +void __vgic_v3_restore_state(struct kvm_vcpu *vcpu); | |
| 61 | + | |
| 62 | +void __timer_save_state(struct kvm_vcpu *vcpu); | |
| 63 | +void __timer_restore_state(struct kvm_vcpu *vcpu); | |
| 64 | + | |
| 65 | +void __sysreg_save_state(struct kvm_cpu_context *ctxt); | |
| 66 | +void __sysreg_restore_state(struct kvm_cpu_context *ctxt); | |
| 67 | +void __sysreg32_save_state(struct kvm_vcpu *vcpu); | |
| 68 | +void __sysreg32_restore_state(struct kvm_vcpu *vcpu); | |
| 69 | + | |
| 70 | +void __debug_save_state(struct kvm_vcpu *vcpu, | |
| 71 | + struct kvm_guest_debug_arch *dbg, | |
| 72 | + struct kvm_cpu_context *ctxt); | |
| 73 | +void __debug_restore_state(struct kvm_vcpu *vcpu, | |
| 74 | + struct kvm_guest_debug_arch *dbg, | |
| 75 | + struct kvm_cpu_context *ctxt); | |
| 76 | +void __debug_cond_save_host_state(struct kvm_vcpu *vcpu); | |
| 77 | +void __debug_cond_restore_host_state(struct kvm_vcpu *vcpu); | |
| 78 | + | |
| 79 | +void __fpsimd_save_state(struct user_fpsimd_state *fp_regs); | |
| 80 | +void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs); | |
| 81 | +static inline bool __fpsimd_enabled(void) | |
| 82 | +{ | |
| 83 | + return !(read_sysreg(cptr_el2) & CPTR_EL2_TFP); | |
| 84 | +} | |
| 85 | + | |
| 86 | +u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt); | |
| 87 | +void __noreturn __hyp_do_panic(unsigned long, ...); | |
| 88 | + | |
| 89 | +#endif /* __ARM64_KVM_HYP_H__ */ | |
| 90 | + | |
| @@ -0,0 +1,175 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include "hyp.h" | |
| 19 | + | |
| 20 | +static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) | |
| 21 | +{ | |
| 22 | + u64 val; | |
| 23 | + | |
| 24 | + /* | |
| 25 | + * We are about to set CPTR_EL2.TFP to trap all floating point | |
| 26 | + * register accesses to EL2, however, the ARM ARM clearly states that | |
| 27 | + * traps are only taken to EL2 if the operation would not otherwise | |
| 28 | + * trap to EL1. Therefore, always make sure that for 32-bit guests, | |
| 29 | + * we set FPEXC.EN to prevent traps to EL1, when setting the TFP bit. | |
| 30 | + */ | |
| 31 | + val = vcpu->arch.hcr_el2; | |
| 32 | + if (!(val & HCR_RW)) { | |
| 33 | + write_sysreg(1 << 30, fpexc32_el2); | |
| 34 | + isb(); | |
| 35 | + } | |
| 36 | + write_sysreg(val, hcr_el2); | |
| 37 | + /* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */ | |
| 38 | + write_sysreg(1 << 15, hstr_el2); | |
| 39 | + write_sysreg(CPTR_EL2_TTA | CPTR_EL2_TFP, cptr_el2); | |
| 40 | + write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2); | |
| 41 | +} | |
| 42 | + | |
| 43 | +static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) | |
| 44 | +{ | |
| 45 | + write_sysreg(HCR_RW, hcr_el2); | |
| 46 | + write_sysreg(0, hstr_el2); | |
| 47 | + write_sysreg(read_sysreg(mdcr_el2) & MDCR_EL2_HPMN_MASK, mdcr_el2); | |
| 48 | + write_sysreg(0, cptr_el2); | |
| 49 | +} | |
| 50 | + | |
| 51 | +static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) | |
| 52 | +{ | |
| 53 | + struct kvm *kvm = kern_hyp_va(vcpu->kvm); | |
| 54 | + write_sysreg(kvm->arch.vttbr, vttbr_el2); | |
| 55 | +} | |
| 56 | + | |
| 57 | +static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) | |
| 58 | +{ | |
| 59 | + write_sysreg(0, vttbr_el2); | |
| 60 | +} | |
| 61 | + | |
| 62 | +static hyp_alternate_select(__vgic_call_save_state, | |
| 63 | + __vgic_v2_save_state, __vgic_v3_save_state, | |
| 64 | + ARM64_HAS_SYSREG_GIC_CPUIF); | |
| 65 | + | |
| 66 | +static hyp_alternate_select(__vgic_call_restore_state, | |
| 67 | + __vgic_v2_restore_state, __vgic_v3_restore_state, | |
| 68 | + ARM64_HAS_SYSREG_GIC_CPUIF); | |
| 69 | + | |
| 70 | +static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu) | |
| 71 | +{ | |
| 72 | + __vgic_call_save_state()(vcpu); | |
| 73 | + write_sysreg(read_sysreg(hcr_el2) & ~HCR_INT_OVERRIDE, hcr_el2); | |
| 74 | +} | |
| 75 | + | |
| 76 | +static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu) | |
| 77 | +{ | |
| 78 | + u64 val; | |
| 79 | + | |
| 80 | + val = read_sysreg(hcr_el2); | |
| 81 | + val |= HCR_INT_OVERRIDE; | |
| 82 | + val |= vcpu->arch.irq_lines; | |
| 83 | + write_sysreg(val, hcr_el2); | |
| 84 | + | |
| 85 | + __vgic_call_restore_state()(vcpu); | |
| 86 | +} | |
| 87 | + | |
| 88 | +static int __hyp_text __guest_run(struct kvm_vcpu *vcpu) | |
| 89 | +{ | |
| 90 | + struct kvm_cpu_context *host_ctxt; | |
| 91 | + struct kvm_cpu_context *guest_ctxt; | |
| 92 | + bool fp_enabled; | |
| 93 | + u64 exit_code; | |
| 94 | + | |
| 95 | + vcpu = kern_hyp_va(vcpu); | |
| 96 | + write_sysreg(vcpu, tpidr_el2); | |
| 97 | + | |
| 98 | + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context); | |
| 99 | + guest_ctxt = &vcpu->arch.ctxt; | |
| 100 | + | |
| 101 | + __sysreg_save_state(host_ctxt); | |
| 102 | + __debug_cond_save_host_state(vcpu); | |
| 103 | + | |
| 104 | + __activate_traps(vcpu); | |
| 105 | + __activate_vm(vcpu); | |
| 106 | + | |
| 107 | + __vgic_restore_state(vcpu); | |
| 108 | + __timer_restore_state(vcpu); | |
| 109 | + | |
| 110 | + /* | |
| 111 | + * We must restore the 32-bit state before the sysregs, thanks | |
| 112 | + * to Cortex-A57 erratum #852523. | |
| 113 | + */ | |
| 114 | + __sysreg32_restore_state(vcpu); | |
| 115 | + __sysreg_restore_state(guest_ctxt); | |
| 116 | + __debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt); | |
| 117 | + | |
| 118 | + /* Jump in the fire! */ | |
| 119 | + exit_code = __guest_enter(vcpu, host_ctxt); | |
| 120 | + /* And we're baaack! */ | |
| 121 | + | |
| 122 | + fp_enabled = __fpsimd_enabled(); | |
| 123 | + | |
| 124 | + __sysreg_save_state(guest_ctxt); | |
| 125 | + __sysreg32_save_state(vcpu); | |
| 126 | + __timer_save_state(vcpu); | |
| 127 | + __vgic_save_state(vcpu); | |
| 128 | + | |
| 129 | + __deactivate_traps(vcpu); | |
| 130 | + __deactivate_vm(vcpu); | |
| 131 | + | |
| 132 | + __sysreg_restore_state(host_ctxt); | |
| 133 | + | |
| 134 | + if (fp_enabled) { | |
| 135 | + __fpsimd_save_state(&guest_ctxt->gp_regs.fp_regs); | |
| 136 | + __fpsimd_restore_state(&host_ctxt->gp_regs.fp_regs); | |
| 137 | + } | |
| 138 | + | |
| 139 | + __debug_save_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt); | |
| 140 | + __debug_cond_restore_host_state(vcpu); | |
| 141 | + | |
| 142 | + return exit_code; | |
| 143 | +} | |
| 144 | + | |
| 145 | +__alias(__guest_run) int __kvm_vcpu_run(struct kvm_vcpu *vcpu); | |
| 146 | + | |
| 147 | +static const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n"; | |
| 148 | + | |
| 149 | +void __hyp_text __noreturn __hyp_panic(void) | |
| 150 | +{ | |
| 151 | + unsigned long str_va = (unsigned long)__hyp_panic_string; | |
| 152 | + u64 spsr = read_sysreg(spsr_el2); | |
| 153 | + u64 elr = read_sysreg(elr_el2); | |
| 154 | + u64 par = read_sysreg(par_el1); | |
| 155 | + | |
| 156 | + if (read_sysreg(vttbr_el2)) { | |
| 157 | + struct kvm_vcpu *vcpu; | |
| 158 | + struct kvm_cpu_context *host_ctxt; | |
| 159 | + | |
| 160 | + vcpu = (struct kvm_vcpu *)read_sysreg(tpidr_el2); | |
| 161 | + host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context); | |
| 162 | + __deactivate_traps(vcpu); | |
| 163 | + __deactivate_vm(vcpu); | |
| 164 | + __sysreg_restore_state(host_ctxt); | |
| 165 | + } | |
| 166 | + | |
| 167 | + /* Call panic for real */ | |
| 168 | + __hyp_do_panic(hyp_kern_va(str_va), | |
| 169 | + spsr, elr, | |
| 170 | + read_sysreg(esr_el2), read_sysreg(far_el2), | |
| 171 | + read_sysreg(hpfar_el2), par, | |
| 172 | + (void *)read_sysreg(tpidr_el2)); | |
| 173 | + | |
| 174 | + unreachable(); | |
| 175 | +} | |
| @@ -0,0 +1,138 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2012-2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/compiler.h> | |
| 19 | +#include <linux/kvm_host.h> | |
| 20 | + | |
| 21 | +#include <asm/kvm_asm.h> | |
| 22 | +#include <asm/kvm_mmu.h> | |
| 23 | + | |
| 24 | +#include "hyp.h" | |
| 25 | + | |
| 26 | +/* ctxt is already in the HYP VA space */ | |
| 27 | +void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt) | |
| 28 | +{ | |
| 29 | + ctxt->sys_regs[MPIDR_EL1] = read_sysreg(vmpidr_el2); | |
| 30 | + ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1); | |
| 31 | + ctxt->sys_regs[SCTLR_EL1] = read_sysreg(sctlr_el1); | |
| 32 | + ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1); | |
| 33 | + ctxt->sys_regs[CPACR_EL1] = read_sysreg(cpacr_el1); | |
| 34 | + ctxt->sys_regs[TTBR0_EL1] = read_sysreg(ttbr0_el1); | |
| 35 | + ctxt->sys_regs[TTBR1_EL1] = read_sysreg(ttbr1_el1); | |
| 36 | + ctxt->sys_regs[TCR_EL1] = read_sysreg(tcr_el1); | |
| 37 | + ctxt->sys_regs[ESR_EL1] = read_sysreg(esr_el1); | |
| 38 | + ctxt->sys_regs[AFSR0_EL1] = read_sysreg(afsr0_el1); | |
| 39 | + ctxt->sys_regs[AFSR1_EL1] = read_sysreg(afsr1_el1); | |
| 40 | + ctxt->sys_regs[FAR_EL1] = read_sysreg(far_el1); | |
| 41 | + ctxt->sys_regs[MAIR_EL1] = read_sysreg(mair_el1); | |
| 42 | + ctxt->sys_regs[VBAR_EL1] = read_sysreg(vbar_el1); | |
| 43 | + ctxt->sys_regs[CONTEXTIDR_EL1] = read_sysreg(contextidr_el1); | |
| 44 | + ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0); | |
| 45 | + ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0); | |
| 46 | + ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1); | |
| 47 | + ctxt->sys_regs[AMAIR_EL1] = read_sysreg(amair_el1); | |
| 48 | + ctxt->sys_regs[CNTKCTL_EL1] = read_sysreg(cntkctl_el1); | |
| 49 | + ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); | |
| 50 | + ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1); | |
| 51 | + | |
| 52 | + ctxt->gp_regs.regs.sp = read_sysreg(sp_el0); | |
| 53 | + ctxt->gp_regs.regs.pc = read_sysreg(elr_el2); | |
| 54 | + ctxt->gp_regs.regs.pstate = read_sysreg(spsr_el2); | |
| 55 | + ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1); | |
| 56 | + ctxt->gp_regs.elr_el1 = read_sysreg(elr_el1); | |
| 57 | + ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg(spsr_el1); | |
| 58 | +} | |
| 59 | + | |
| 60 | +void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt) | |
| 61 | +{ | |
| 62 | + write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2); | |
| 63 | + write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1); | |
| 64 | + write_sysreg(ctxt->sys_regs[SCTLR_EL1], sctlr_el1); | |
| 65 | + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1); | |
| 66 | + write_sysreg(ctxt->sys_regs[CPACR_EL1], cpacr_el1); | |
| 67 | + write_sysreg(ctxt->sys_regs[TTBR0_EL1], ttbr0_el1); | |
| 68 | + write_sysreg(ctxt->sys_regs[TTBR1_EL1], ttbr1_el1); | |
| 69 | + write_sysreg(ctxt->sys_regs[TCR_EL1], tcr_el1); | |
| 70 | + write_sysreg(ctxt->sys_regs[ESR_EL1], esr_el1); | |
| 71 | + write_sysreg(ctxt->sys_regs[AFSR0_EL1], afsr0_el1); | |
| 72 | + write_sysreg(ctxt->sys_regs[AFSR1_EL1], afsr1_el1); | |
| 73 | + write_sysreg(ctxt->sys_regs[FAR_EL1], far_el1); | |
| 74 | + write_sysreg(ctxt->sys_regs[MAIR_EL1], mair_el1); | |
| 75 | + write_sysreg(ctxt->sys_regs[VBAR_EL1], vbar_el1); | |
| 76 | + write_sysreg(ctxt->sys_regs[CONTEXTIDR_EL1], contextidr_el1); | |
| 77 | + write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0); | |
| 78 | + write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0); | |
| 79 | + write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1); | |
| 80 | + write_sysreg(ctxt->sys_regs[AMAIR_EL1], amair_el1); | |
| 81 | + write_sysreg(ctxt->sys_regs[CNTKCTL_EL1], cntkctl_el1); | |
| 82 | + write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1); | |
| 83 | + write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1); | |
| 84 | + | |
| 85 | + write_sysreg(ctxt->gp_regs.regs.sp, sp_el0); | |
| 86 | + write_sysreg(ctxt->gp_regs.regs.pc, elr_el2); | |
| 87 | + write_sysreg(ctxt->gp_regs.regs.pstate, spsr_el2); | |
| 88 | + write_sysreg(ctxt->gp_regs.sp_el1, sp_el1); | |
| 89 | + write_sysreg(ctxt->gp_regs.elr_el1, elr_el1); | |
| 90 | + write_sysreg(ctxt->gp_regs.spsr[KVM_SPSR_EL1], spsr_el1); | |
| 91 | +} | |
| 92 | + | |
| 93 | +void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) | |
| 94 | +{ | |
| 95 | + u64 *spsr, *sysreg; | |
| 96 | + | |
| 97 | + if (read_sysreg(hcr_el2) & HCR_RW) | |
| 98 | + return; | |
| 99 | + | |
| 100 | + spsr = vcpu->arch.ctxt.gp_regs.spsr; | |
| 101 | + sysreg = vcpu->arch.ctxt.sys_regs; | |
| 102 | + | |
| 103 | + spsr[KVM_SPSR_ABT] = read_sysreg(spsr_abt); | |
| 104 | + spsr[KVM_SPSR_UND] = read_sysreg(spsr_und); | |
| 105 | + spsr[KVM_SPSR_IRQ] = read_sysreg(spsr_irq); | |
| 106 | + spsr[KVM_SPSR_FIQ] = read_sysreg(spsr_fiq); | |
| 107 | + | |
| 108 | + sysreg[DACR32_EL2] = read_sysreg(dacr32_el2); | |
| 109 | + sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2); | |
| 110 | + | |
| 111 | + if (__fpsimd_enabled()) | |
| 112 | + sysreg[FPEXC32_EL2] = read_sysreg(fpexc32_el2); | |
| 113 | + | |
| 114 | + if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY) | |
| 115 | + sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2); | |
| 116 | +} | |
| 117 | + | |
| 118 | +void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) | |
| 119 | +{ | |
| 120 | + u64 *spsr, *sysreg; | |
| 121 | + | |
| 122 | + if (read_sysreg(hcr_el2) & HCR_RW) | |
| 123 | + return; | |
| 124 | + | |
| 125 | + spsr = vcpu->arch.ctxt.gp_regs.spsr; | |
| 126 | + sysreg = vcpu->arch.ctxt.sys_regs; | |
| 127 | + | |
| 128 | + write_sysreg(spsr[KVM_SPSR_ABT], spsr_abt); | |
| 129 | + write_sysreg(spsr[KVM_SPSR_UND], spsr_und); | |
| 130 | + write_sysreg(spsr[KVM_SPSR_IRQ], spsr_irq); | |
| 131 | + write_sysreg(spsr[KVM_SPSR_FIQ], spsr_fiq); | |
| 132 | + | |
| 133 | + write_sysreg(sysreg[DACR32_EL2], dacr32_el2); | |
| 134 | + write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2); | |
| 135 | + | |
| 136 | + if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY) | |
| 137 | + write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2); | |
| 138 | +} | |
| @@ -0,0 +1,71 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2012-2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <clocksource/arm_arch_timer.h> | |
| 19 | +#include <linux/compiler.h> | |
| 20 | +#include <linux/kvm_host.h> | |
| 21 | + | |
| 22 | +#include <asm/kvm_mmu.h> | |
| 23 | + | |
| 24 | +#include "hyp.h" | |
| 25 | + | |
| 26 | +/* vcpu is already in the HYP VA space */ | |
| 27 | +void __hyp_text __timer_save_state(struct kvm_vcpu *vcpu) | |
| 28 | +{ | |
| 29 | + struct kvm *kvm = kern_hyp_va(vcpu->kvm); | |
| 30 | + struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu; | |
| 31 | + u64 val; | |
| 32 | + | |
| 33 | + if (kvm->arch.timer.enabled) { | |
| 34 | + timer->cntv_ctl = read_sysreg(cntv_ctl_el0); | |
| 35 | + timer->cntv_cval = read_sysreg(cntv_cval_el0); | |
| 36 | + } | |
| 37 | + | |
| 38 | + /* Disable the virtual timer */ | |
| 39 | + write_sysreg(0, cntv_ctl_el0); | |
| 40 | + | |
| 41 | + /* Allow physical timer/counter access for the host */ | |
| 42 | + val = read_sysreg(cnthctl_el2); | |
| 43 | + val |= CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN; | |
| 44 | + write_sysreg(val, cnthctl_el2); | |
| 45 | + | |
| 46 | + /* Clear cntvoff for the host */ | |
| 47 | + write_sysreg(0, cntvoff_el2); | |
| 48 | +} | |
| 49 | + | |
| 50 | +void __hyp_text __timer_restore_state(struct kvm_vcpu *vcpu) | |
| 51 | +{ | |
| 52 | + struct kvm *kvm = kern_hyp_va(vcpu->kvm); | |
| 53 | + struct arch_timer_cpu *timer = &vcpu->arch.timer_cpu; | |
| 54 | + u64 val; | |
| 55 | + | |
| 56 | + /* | |
| 57 | + * Disallow physical timer access for the guest | |
| 58 | + * Physical counter access is allowed | |
| 59 | + */ | |
| 60 | + val = read_sysreg(cnthctl_el2); | |
| 61 | + val &= ~CNTHCTL_EL1PCEN; | |
| 62 | + val |= CNTHCTL_EL1PCTEN; | |
| 63 | + write_sysreg(val, cnthctl_el2); | |
| 64 | + | |
| 65 | + if (kvm->arch.timer.enabled) { | |
| 66 | + write_sysreg(kvm->arch.timer.cntvoff, cntvoff_el2); | |
| 67 | + write_sysreg(timer->cntv_cval, cntv_cval_el0); | |
| 68 | + isb(); | |
| 69 | + write_sysreg(timer->cntv_ctl, cntv_ctl_el0); | |
| 70 | + } | |
| 71 | +} | |
| @@ -0,0 +1,80 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include "hyp.h" | |
| 19 | + | |
| 20 | +static void __hyp_text __tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) | |
| 21 | +{ | |
| 22 | + dsb(ishst); | |
| 23 | + | |
| 24 | + /* Switch to requested VMID */ | |
| 25 | + kvm = kern_hyp_va(kvm); | |
| 26 | + write_sysreg(kvm->arch.vttbr, vttbr_el2); | |
| 27 | + isb(); | |
| 28 | + | |
| 29 | + /* | |
| 30 | + * We could do so much better if we had the VA as well. | |
| 31 | + * Instead, we invalidate Stage-2 for this IPA, and the | |
| 32 | + * whole of Stage-1. Weep... | |
| 33 | + */ | |
| 34 | + ipa >>= 12; | |
| 35 | + asm volatile("tlbi ipas2e1is, %0" : : "r" (ipa)); | |
| 36 | + | |
| 37 | + /* | |
| 38 | + * We have to ensure completion of the invalidation at Stage-2, | |
| 39 | + * since a table walk on another CPU could refill a TLB with a | |
| 40 | + * complete (S1 + S2) walk based on the old Stage-2 mapping if | |
| 41 | + * the Stage-1 invalidation happened first. | |
| 42 | + */ | |
| 43 | + dsb(ish); | |
| 44 | + asm volatile("tlbi vmalle1is" : : ); | |
| 45 | + dsb(ish); | |
| 46 | + isb(); | |
| 47 | + | |
| 48 | + write_sysreg(0, vttbr_el2); | |
| 49 | +} | |
| 50 | + | |
| 51 | +__alias(__tlb_flush_vmid_ipa) void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, | |
| 52 | + phys_addr_t ipa); | |
| 53 | + | |
| 54 | +static void __hyp_text __tlb_flush_vmid(struct kvm *kvm) | |
| 55 | +{ | |
| 56 | + dsb(ishst); | |
| 57 | + | |
| 58 | + /* Switch to requested VMID */ | |
| 59 | + kvm = kern_hyp_va(kvm); | |
| 60 | + write_sysreg(kvm->arch.vttbr, vttbr_el2); | |
| 61 | + isb(); | |
| 62 | + | |
| 63 | + asm volatile("tlbi vmalls12e1is" : : ); | |
| 64 | + dsb(ish); | |
| 65 | + isb(); | |
| 66 | + | |
| 67 | + write_sysreg(0, vttbr_el2); | |
| 68 | +} | |
| 69 | + | |
| 70 | +__alias(__tlb_flush_vmid) void __kvm_tlb_flush_vmid(struct kvm *kvm); | |
| 71 | + | |
| 72 | +static void __hyp_text __tlb_flush_vm_context(void) | |
| 73 | +{ | |
| 74 | + dsb(ishst); | |
| 75 | + asm volatile("tlbi alle1is \n" | |
| 76 | + "ic ialluis ": : ); | |
| 77 | + dsb(ish); | |
| 78 | +} | |
| 79 | + | |
| 80 | +__alias(__tlb_flush_vm_context) void __kvm_flush_vm_context(void); | |
| @@ -0,0 +1,84 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2012-2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/compiler.h> | |
| 19 | +#include <linux/irqchip/arm-gic.h> | |
| 20 | +#include <linux/kvm_host.h> | |
| 21 | + | |
| 22 | +#include <asm/kvm_mmu.h> | |
| 23 | + | |
| 24 | +#include "hyp.h" | |
| 25 | + | |
| 26 | +/* vcpu is already in the HYP VA space */ | |
| 27 | +void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu) | |
| 28 | +{ | |
| 29 | + struct kvm *kvm = kern_hyp_va(vcpu->kvm); | |
| 30 | + struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; | |
| 31 | + struct vgic_dist *vgic = &kvm->arch.vgic; | |
| 32 | + void __iomem *base = kern_hyp_va(vgic->vctrl_base); | |
| 33 | + u32 eisr0, eisr1, elrsr0, elrsr1; | |
| 34 | + int i, nr_lr; | |
| 35 | + | |
| 36 | + if (!base) | |
| 37 | + return; | |
| 38 | + | |
| 39 | + nr_lr = vcpu->arch.vgic_cpu.nr_lr; | |
| 40 | + cpu_if->vgic_vmcr = readl_relaxed(base + GICH_VMCR); | |
| 41 | + cpu_if->vgic_misr = readl_relaxed(base + GICH_MISR); | |
| 42 | + eisr0 = readl_relaxed(base + GICH_EISR0); | |
| 43 | + elrsr0 = readl_relaxed(base + GICH_ELRSR0); | |
| 44 | + if (unlikely(nr_lr > 32)) { | |
| 45 | + eisr1 = readl_relaxed(base + GICH_EISR1); | |
| 46 | + elrsr1 = readl_relaxed(base + GICH_ELRSR1); | |
| 47 | + } else { | |
| 48 | + eisr1 = elrsr1 = 0; | |
| 49 | + } | |
| 50 | +#ifdef CONFIG_CPU_BIG_ENDIAN | |
| 51 | + cpu_if->vgic_eisr = ((u64)eisr0 << 32) | eisr1; | |
| 52 | + cpu_if->vgic_elrsr = ((u64)elrsr0 << 32) | elrsr1; | |
| 53 | +#else | |
| 54 | + cpu_if->vgic_eisr = ((u64)eisr1 << 32) | eisr0; | |
| 55 | + cpu_if->vgic_elrsr = ((u64)elrsr1 << 32) | elrsr0; | |
| 56 | +#endif | |
| 57 | + cpu_if->vgic_apr = readl_relaxed(base + GICH_APR); | |
| 58 | + | |
| 59 | + writel_relaxed(0, base + GICH_HCR); | |
| 60 | + | |
| 61 | + for (i = 0; i < nr_lr; i++) | |
| 62 | + cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4)); | |
| 63 | +} | |
| 64 | + | |
| 65 | +/* vcpu is already in the HYP VA space */ | |
| 66 | +void __hyp_text __vgic_v2_restore_state(struct kvm_vcpu *vcpu) | |
| 67 | +{ | |
| 68 | + struct kvm *kvm = kern_hyp_va(vcpu->kvm); | |
| 69 | + struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; | |
| 70 | + struct vgic_dist *vgic = &kvm->arch.vgic; | |
| 71 | + void __iomem *base = kern_hyp_va(vgic->vctrl_base); | |
| 72 | + int i, nr_lr; | |
| 73 | + | |
| 74 | + if (!base) | |
| 75 | + return; | |
| 76 | + | |
| 77 | + writel_relaxed(cpu_if->vgic_hcr, base + GICH_HCR); | |
| 78 | + writel_relaxed(cpu_if->vgic_vmcr, base + GICH_VMCR); | |
| 79 | + writel_relaxed(cpu_if->vgic_apr, base + GICH_APR); | |
| 80 | + | |
| 81 | + nr_lr = vcpu->arch.vgic_cpu.nr_lr; | |
| 82 | + for (i = 0; i < nr_lr; i++) | |
| 83 | + writel_relaxed(cpu_if->vgic_lr[i], base + GICH_LR0 + (i * 4)); | |
| 84 | +} | |
| @@ -0,0 +1,228 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2012-2015 - ARM Ltd | |
| 3 | + * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | + * | |
| 5 | + * This program is free software; you can redistribute it and/or modify | |
| 6 | + * it under the terms of the GNU General Public License version 2 as | |
| 7 | + * published by the Free Software Foundation. | |
| 8 | + * | |
| 9 | + * This program is distributed in the hope that it will be useful, | |
| 10 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | + * GNU General Public License for more details. | |
| 13 | + * | |
| 14 | + * You should have received a copy of the GNU General Public License | |
| 15 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | + */ | |
| 17 | + | |
| 18 | +#include <linux/compiler.h> | |
| 19 | +#include <linux/irqchip/arm-gic-v3.h> | |
| 20 | +#include <linux/kvm_host.h> | |
| 21 | + | |
| 22 | +#include <asm/kvm_mmu.h> | |
| 23 | + | |
| 24 | +#include "hyp.h" | |
| 25 | + | |
| 26 | +#define vtr_to_max_lr_idx(v) ((v) & 0xf) | |
| 27 | +#define vtr_to_nr_pri_bits(v) (((u32)(v) >> 29) + 1) | |
| 28 | + | |
| 29 | +#define read_gicreg(r) \ | |
| 30 | + ({ \ | |
| 31 | + u64 reg; \ | |
| 32 | + asm volatile("mrs_s %0, " __stringify(r) : "=r" (reg)); \ | |
| 33 | + reg; \ | |
| 34 | + }) | |
| 35 | + | |
| 36 | +#define write_gicreg(v,r) \ | |
| 37 | + do { \ | |
| 38 | + u64 __val = (v); \ | |
| 39 | + asm volatile("msr_s " __stringify(r) ", %0" : : "r" (__val));\ | |
| 40 | + } while (0) | |
| 41 | + | |
| 42 | +/* vcpu is already in the HYP VA space */ | |
| 43 | +void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu) | |
| 44 | +{ | |
| 45 | + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; | |
| 46 | + u64 val; | |
| 47 | + u32 max_lr_idx, nr_pri_bits; | |
| 48 | + | |
| 49 | + /* | |
| 50 | + * Make sure stores to the GIC via the memory mapped interface | |
| 51 | + * are now visible to the system register interface. | |
| 52 | + */ | |
| 53 | + dsb(st); | |
| 54 | + | |
| 55 | + cpu_if->vgic_vmcr = read_gicreg(ICH_VMCR_EL2); | |
| 56 | + cpu_if->vgic_misr = read_gicreg(ICH_MISR_EL2); | |
| 57 | + cpu_if->vgic_eisr = read_gicreg(ICH_EISR_EL2); | |
| 58 | + cpu_if->vgic_elrsr = read_gicreg(ICH_ELSR_EL2); | |
| 59 | + | |
| 60 | + write_gicreg(0, ICH_HCR_EL2); | |
| 61 | + val = read_gicreg(ICH_VTR_EL2); | |
| 62 | + max_lr_idx = vtr_to_max_lr_idx(val); | |
| 63 | + nr_pri_bits = vtr_to_nr_pri_bits(val); | |
| 64 | + | |
| 65 | + switch (max_lr_idx) { | |
| 66 | + case 15: | |
| 67 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(15)] = read_gicreg(ICH_LR15_EL2); | |
| 68 | + case 14: | |
| 69 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(14)] = read_gicreg(ICH_LR14_EL2); | |
| 70 | + case 13: | |
| 71 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(13)] = read_gicreg(ICH_LR13_EL2); | |
| 72 | + case 12: | |
| 73 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(12)] = read_gicreg(ICH_LR12_EL2); | |
| 74 | + case 11: | |
| 75 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(11)] = read_gicreg(ICH_LR11_EL2); | |
| 76 | + case 10: | |
| 77 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(10)] = read_gicreg(ICH_LR10_EL2); | |
| 78 | + case 9: | |
| 79 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(9)] = read_gicreg(ICH_LR9_EL2); | |
| 80 | + case 8: | |
| 81 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(8)] = read_gicreg(ICH_LR8_EL2); | |
| 82 | + case 7: | |
| 83 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(7)] = read_gicreg(ICH_LR7_EL2); | |
| 84 | + case 6: | |
| 85 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(6)] = read_gicreg(ICH_LR6_EL2); | |
| 86 | + case 5: | |
| 87 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(5)] = read_gicreg(ICH_LR5_EL2); | |
| 88 | + case 4: | |
| 89 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(4)] = read_gicreg(ICH_LR4_EL2); | |
| 90 | + case 3: | |
| 91 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(3)] = read_gicreg(ICH_LR3_EL2); | |
| 92 | + case 2: | |
| 93 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(2)] = read_gicreg(ICH_LR2_EL2); | |
| 94 | + case 1: | |
| 95 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(1)] = read_gicreg(ICH_LR1_EL2); | |
| 96 | + case 0: | |
| 97 | + cpu_if->vgic_lr[VGIC_V3_LR_INDEX(0)] = read_gicreg(ICH_LR0_EL2); | |
| 98 | + } | |
| 99 | + | |
| 100 | + switch (nr_pri_bits) { | |
| 101 | + case 7: | |
| 102 | + cpu_if->vgic_ap0r[3] = read_gicreg(ICH_AP0R3_EL2); | |
| 103 | + cpu_if->vgic_ap0r[2] = read_gicreg(ICH_AP0R2_EL2); | |
| 104 | + case 6: | |
| 105 | + cpu_if->vgic_ap0r[1] = read_gicreg(ICH_AP0R1_EL2); | |
| 106 | + default: | |
| 107 | + cpu_if->vgic_ap0r[0] = read_gicreg(ICH_AP0R0_EL2); | |
| 108 | + } | |
| 109 | + | |
| 110 | + switch (nr_pri_bits) { | |
| 111 | + case 7: | |
| 112 | + cpu_if->vgic_ap1r[3] = read_gicreg(ICH_AP1R3_EL2); | |
| 113 | + cpu_if->vgic_ap1r[2] = read_gicreg(ICH_AP1R2_EL2); | |
| 114 | + case 6: | |
| 115 | + cpu_if->vgic_ap1r[1] = read_gicreg(ICH_AP1R1_EL2); | |
| 116 | + default: | |
| 117 | + cpu_if->vgic_ap1r[0] = read_gicreg(ICH_AP1R0_EL2); | |
| 118 | + } | |
| 119 | + | |
| 120 | + val = read_gicreg(ICC_SRE_EL2); | |
| 121 | + write_gicreg(val | ICC_SRE_EL2_ENABLE, ICC_SRE_EL2); | |
| 122 | + isb(); /* Make sure ENABLE is set at EL2 before setting SRE at EL1 */ | |
| 123 | + write_gicreg(1, ICC_SRE_EL1); | |
| 124 | +} | |
| 125 | + | |
| 126 | +void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu) | |
| 127 | +{ | |
| 128 | + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; | |
| 129 | + u64 val; | |
| 130 | + u32 max_lr_idx, nr_pri_bits; | |
| 131 | + | |
| 132 | + /* | |
| 133 | + * VFIQEn is RES1 if ICC_SRE_EL1.SRE is 1. This causes a | |
| 134 | + * Group0 interrupt (as generated in GICv2 mode) to be | |
| 135 | + * delivered as a FIQ to the guest, with potentially fatal | |
| 136 | + * consequences. So we must make sure that ICC_SRE_EL1 has | |
| 137 | + * been actually programmed with the value we want before | |
| 138 | + * starting to mess with the rest of the GIC. | |
| 139 | + */ | |
| 140 | + write_gicreg(cpu_if->vgic_sre, ICC_SRE_EL1); | |
| 141 | + isb(); | |
| 142 | + | |
| 143 | + write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2); | |
| 144 | + write_gicreg(cpu_if->vgic_vmcr, ICH_VMCR_EL2); | |
| 145 | + | |
| 146 | + val = read_gicreg(ICH_VTR_EL2); | |
| 147 | + max_lr_idx = vtr_to_max_lr_idx(val); | |
| 148 | + nr_pri_bits = vtr_to_nr_pri_bits(val); | |
| 149 | + | |
| 150 | + switch (nr_pri_bits) { | |
| 151 | + case 7: | |
| 152 | + write_gicreg(cpu_if->vgic_ap1r[3], ICH_AP1R3_EL2); | |
| 153 | + write_gicreg(cpu_if->vgic_ap1r[2], ICH_AP1R2_EL2); | |
| 154 | + case 6: | |
| 155 | + write_gicreg(cpu_if->vgic_ap1r[1], ICH_AP1R1_EL2); | |
| 156 | + default: | |
| 157 | + write_gicreg(cpu_if->vgic_ap1r[0], ICH_AP1R0_EL2); | |
| 158 | + } | |
| 159 | + | |
| 160 | + switch (nr_pri_bits) { | |
| 161 | + case 7: | |
| 162 | + write_gicreg(cpu_if->vgic_ap0r[3], ICH_AP0R3_EL2); | |
| 163 | + write_gicreg(cpu_if->vgic_ap0r[2], ICH_AP0R2_EL2); | |
| 164 | + case 6: | |
| 165 | + write_gicreg(cpu_if->vgic_ap0r[1], ICH_AP0R1_EL2); | |
| 166 | + default: | |
| 167 | + write_gicreg(cpu_if->vgic_ap0r[0], ICH_AP0R0_EL2); | |
| 168 | + } | |
| 169 | + | |
| 170 | + switch (max_lr_idx) { | |
| 171 | + case 15: | |
| 172 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(15)], ICH_LR15_EL2); | |
| 173 | + case 14: | |
| 174 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(14)], ICH_LR14_EL2); | |
| 175 | + case 13: | |
| 176 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(13)], ICH_LR13_EL2); | |
| 177 | + case 12: | |
| 178 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(12)], ICH_LR12_EL2); | |
| 179 | + case 11: | |
| 180 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(11)], ICH_LR11_EL2); | |
| 181 | + case 10: | |
| 182 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(10)], ICH_LR10_EL2); | |
| 183 | + case 9: | |
| 184 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(9)], ICH_LR9_EL2); | |
| 185 | + case 8: | |
| 186 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(8)], ICH_LR8_EL2); | |
| 187 | + case 7: | |
| 188 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(7)], ICH_LR7_EL2); | |
| 189 | + case 6: | |
| 190 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(6)], ICH_LR6_EL2); | |
| 191 | + case 5: | |
| 192 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(5)], ICH_LR5_EL2); | |
| 193 | + case 4: | |
| 194 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(4)], ICH_LR4_EL2); | |
| 195 | + case 3: | |
| 196 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(3)], ICH_LR3_EL2); | |
| 197 | + case 2: | |
| 198 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(2)], ICH_LR2_EL2); | |
| 199 | + case 1: | |
| 200 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(1)], ICH_LR1_EL2); | |
| 201 | + case 0: | |
| 202 | + write_gicreg(cpu_if->vgic_lr[VGIC_V3_LR_INDEX(0)], ICH_LR0_EL2); | |
| 203 | + } | |
| 204 | + | |
| 205 | + /* | |
| 206 | + * Ensures that the above will have reached the | |
| 207 | + * (re)distributors. This ensure the guest will read the | |
| 208 | + * correct values from the memory-mapped interface. | |
| 209 | + */ | |
| 210 | + isb(); | |
| 211 | + dsb(sy); | |
| 212 | + | |
| 213 | + /* | |
| 214 | + * Prevent the guest from touching the GIC system registers if | |
| 215 | + * SRE isn't enabled for GICv3 emulation. | |
| 216 | + */ | |
| 217 | + if (!cpu_if->vgic_sre) { | |
| 218 | + write_gicreg(read_gicreg(ICC_SRE_EL2) & ~ICC_SRE_EL2_ENABLE, | |
| 219 | + ICC_SRE_EL2); | |
| 220 | + } | |
| 221 | +} | |
| 222 | + | |
| 223 | +static u64 __hyp_text __vgic_v3_read_ich_vtr_el2(void) | |
| 224 | +{ | |
| 225 | + return read_gicreg(ICH_VTR_EL2); | |
| 226 | +} | |
| 227 | + | |
| 228 | +__alias(__vgic_v3_read_ich_vtr_el2) u64 __vgic_v3_get_ich_vtr_el2(void); | |
| @@ -29,6 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | #include <asm/debug-monitors.h> |
| 30 | 30 | #include <asm/esr.h> |
| 31 | 31 | #include <asm/kvm_arm.h> |
| 32 | +#include <asm/kvm_asm.h> | |
| 32 | 33 | #include <asm/kvm_coproc.h> |
| 33 | 34 | #include <asm/kvm_emulate.h> |
| 34 | 35 | #include <asm/kvm_host.h> |
| @@ -78,7 +79,7 @@ static u32 get_ccsidr(u32 csselr) · 43 unmodified lines | ||
| 78 | 79 | * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). |
| 79 | 80 | */ |
| 80 | 81 | static bool access_dcsw(struct kvm_vcpu *vcpu, |
| 81 | − const struct sys_reg_params *p, | |
| 82 | + struct sys_reg_params *p, | |
| 82 | 83 | const struct sys_reg_desc *r) |
| 83 | 84 | { |
| 84 | 85 | if (!p->is_write) |
| @@ -94,21 +95,19 @@ static bool access_dcsw(struct kvm_vcpu *vcpu, · 9 unmodified lines | ||
| 94 | 95 | * sys_regs and leave it in complete control of the caches. |
| 95 | 96 | */ |
| 96 | 97 | static bool access_vm_reg(struct kvm_vcpu *vcpu, |
| 97 | − const struct sys_reg_params *p, | |
| 98 | + struct sys_reg_params *p, | |
| 98 | 99 | const struct sys_reg_desc *r) |
| 99 | 100 | { |
| 100 | − unsigned long val; | |
| 101 | 101 | bool was_enabled = vcpu_has_cache_enabled(vcpu); |
| 102 | 102 | |
| 103 | 103 | BUG_ON(!p->is_write); |
| 104 | 104 | |
| 105 | − val = *vcpu_reg(vcpu, p->Rt); | |
| 106 | 105 | if (!p->is_aarch32) { |
| 107 | − vcpu_sys_reg(vcpu, r->reg) = val; | |
| 106 | + vcpu_sys_reg(vcpu, r->reg) = p->regval; | |
| 108 | 107 | } else { |
| 109 | 108 | if (!p->is_32bit) |
| 110 | − vcpu_cp15_64_high(vcpu, r->reg) = val >> 32; | |
| 111 | − vcpu_cp15_64_low(vcpu, r->reg) = val & 0xffffffffUL; | |
| 109 | + vcpu_cp15_64_high(vcpu, r->reg) = upper_32_bits(p->regval); | |
| 110 | + vcpu_cp15_64_low(vcpu, r->reg) = lower_32_bits(p->regval); | |
| 112 | 111 | } |
| 113 | 112 | |
| 114 | 113 | kvm_toggle_cache(vcpu, was_enabled); |
| @@ -122,22 +121,19 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, · 7 unmodified lines | ||
| 122 | 121 | * for both AArch64 and AArch32 accesses. |
| 123 | 122 | */ |
| 124 | 123 | static bool access_gic_sgi(struct kvm_vcpu *vcpu, |
| 125 | − const struct sys_reg_params *p, | |
| 124 | + struct sys_reg_params *p, | |
| 126 | 125 | const struct sys_reg_desc *r) |
| 127 | 126 | { |
| 128 | − u64 val; | |
| 129 | − | |
| 130 | 127 | if (!p->is_write) |
| 131 | 128 | return read_from_write_only(vcpu, p); |
| 132 | 129 | |
| 133 | − val = *vcpu_reg(vcpu, p->Rt); | |
| 134 | − vgic_v3_dispatch_sgi(vcpu, val); | |
| 130 | + vgic_v3_dispatch_sgi(vcpu, p->regval); | |
| 135 | 131 | |
| 136 | 132 | return true; |
| 137 | 133 | } |
| 138 | 134 | |
| 139 | 135 | static bool trap_raz_wi(struct kvm_vcpu *vcpu, |
| 140 | − const struct sys_reg_params *p, | |
| 136 | + struct sys_reg_params *p, | |
| 141 | 137 | const struct sys_reg_desc *r) |
| 142 | 138 | { |
| 143 | 139 | if (p->is_write) |
| @@ -147,19 +143,19 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, · 3 unmodified lines | ||
| 147 | 143 | } |
| 148 | 144 | |
| 149 | 145 | static bool trap_oslsr_el1(struct kvm_vcpu *vcpu, |
| 150 | − const struct sys_reg_params *p, | |
| 146 | + struct sys_reg_params *p, | |
| 151 | 147 | const struct sys_reg_desc *r) |
| 152 | 148 | { |
| 153 | 149 | if (p->is_write) { |
| 154 | 150 | return ignore_write(vcpu, p); |
| 155 | 151 | } else { |
| 156 | − *vcpu_reg(vcpu, p->Rt) = (1 << 3); | |
| 152 | + p->regval = (1 << 3); | |
| 157 | 153 | return true; |
| 158 | 154 | } |
| 159 | 155 | } |
| 160 | 156 | |
| 161 | 157 | static bool trap_dbgauthstatus_el1(struct kvm_vcpu *vcpu, |
| 162 | − const struct sys_reg_params *p, | |
| 158 | + struct sys_reg_params *p, | |
| 163 | 159 | const struct sys_reg_desc *r) |
| 164 | 160 | { |
| 165 | 161 | if (p->is_write) { |
| @@ -167,7 +163,7 @@ static bool trap_dbgauthstatus_el1(struct kvm_vcpu *vcpu, · 1 unmodified line | ||
| 167 | 163 | } else { |
| 168 | 164 | u32 val; |
| 169 | 165 | asm volatile("mrs %0, dbgauthstatus_el1" : "=r" (val)); |
| 170 | − *vcpu_reg(vcpu, p->Rt) = val; | |
| 166 | + p->regval = val; | |
| 171 | 167 | return true; |
| 172 | 168 | } |
| 173 | 169 | } |
| @@ -200,17 +196,17 @@ static bool trap_dbgauthstatus_el1(struct kvm_vcpu *vcpu, · 26 unmodified lines | ||
| 200 | 196 | * now use the debug registers. |
| 201 | 197 | */ |
| 202 | 198 | static bool trap_debug_regs(struct kvm_vcpu *vcpu, |
| 203 | − const struct sys_reg_params *p, | |
| 199 | + struct sys_reg_params *p, | |
| 204 | 200 | const struct sys_reg_desc *r) |
| 205 | 201 | { |
| 206 | 202 | if (p->is_write) { |
| 207 | − vcpu_sys_reg(vcpu, r->reg) = *vcpu_reg(vcpu, p->Rt); | |
| 203 | + vcpu_sys_reg(vcpu, r->reg) = p->regval; | |
| 208 | 204 | vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; |
| 209 | 205 | } else { |
| 210 | − *vcpu_reg(vcpu, p->Rt) = vcpu_sys_reg(vcpu, r->reg); | |
| 206 | + p->regval = vcpu_sys_reg(vcpu, r->reg); | |
| 211 | 207 | } |
| 212 | 208 | |
| 213 | − trace_trap_reg(__func__, r->reg, p->is_write, *vcpu_reg(vcpu, p->Rt)); | |
| 209 | + trace_trap_reg(__func__, r->reg, p->is_write, p->regval); | |
| 214 | 210 | |
| 215 | 211 | return true; |
| 216 | 212 | } |
| @@ -224,11 +220,11 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, · 7 unmodified lines | ||
| 224 | 220 | * All writes will set the KVM_ARM64_DEBUG_DIRTY flag to ensure the |
| 225 | 221 | * hyp.S code switches between host and guest values in future. |
| 226 | 222 | */ |
| 227 | −static inline void reg_to_dbg(struct kvm_vcpu *vcpu, | |
| 228 | − const struct sys_reg_params *p, | |
| 229 | − u64 *dbg_reg) | |
| 223 | +static void reg_to_dbg(struct kvm_vcpu *vcpu, | |
| 224 | + struct sys_reg_params *p, | |
| 225 | + u64 *dbg_reg) | |
| 230 | 226 | { |
| 231 | − u64 val = *vcpu_reg(vcpu, p->Rt); | |
| 227 | + u64 val = p->regval; | |
| 232 | 228 | |
| 233 | 229 | if (p->is_32bit) { |
| 234 | 230 | val &= 0xffffffffUL; |
| @@ -239,21 +235,18 @@ static inline void reg_to_dbg(struct kvm_vcpu *vcpu, · 4 unmodified lines | ||
| 239 | 235 | vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; |
| 240 | 236 | } |
| 241 | 237 | |
| 242 | −static inline void dbg_to_reg(struct kvm_vcpu *vcpu, | |
| 243 | − const struct sys_reg_params *p, | |
| 244 | − u64 *dbg_reg) | |
| 238 | +static void dbg_to_reg(struct kvm_vcpu *vcpu, | |
| 239 | + struct sys_reg_params *p, | |
| 240 | + u64 *dbg_reg) | |
| 245 | 241 | { |
| 246 | − u64 val = *dbg_reg; | |
| 247 | − | |
| 242 | + p->regval = *dbg_reg; | |
| 248 | 243 | if (p->is_32bit) |
| 249 | − val &= 0xffffffffUL; | |
| 250 | − | |
| 251 | − *vcpu_reg(vcpu, p->Rt) = val; | |
| 244 | + p->regval &= 0xffffffffUL; | |
| 252 | 245 | } |
| 253 | 246 | |
| 254 | −static inline bool trap_bvr(struct kvm_vcpu *vcpu, | |
| 255 | − const struct sys_reg_params *p, | |
| 256 | − const struct sys_reg_desc *rd) | |
| 247 | +static bool trap_bvr(struct kvm_vcpu *vcpu, | |
| 248 | + struct sys_reg_params *p, | |
| 249 | + const struct sys_reg_desc *rd) | |
| 257 | 250 | { |
| 258 | 251 | u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg]; |
| 259 | 252 | |
| @@ -287,15 +280,15 @@ static int get_bvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, · 27 unmodified lines | ||
| 287 | 280 | return 0; |
| 288 | 281 | } |
| 289 | 282 | |
| 290 | −static inline void reset_bvr(struct kvm_vcpu *vcpu, | |
| 291 | − const struct sys_reg_desc *rd) | |
| 283 | +static void reset_bvr(struct kvm_vcpu *vcpu, | |
| 284 | + const struct sys_reg_desc *rd) | |
| 292 | 285 | { |
| 293 | 286 | vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg] = rd->val; |
| 294 | 287 | } |
| 295 | 288 | |
| 296 | −static inline bool trap_bcr(struct kvm_vcpu *vcpu, | |
| 297 | − const struct sys_reg_params *p, | |
| 298 | − const struct sys_reg_desc *rd) | |
| 289 | +static bool trap_bcr(struct kvm_vcpu *vcpu, | |
| 290 | + struct sys_reg_params *p, | |
| 291 | + const struct sys_reg_desc *rd) | |
| 299 | 292 | { |
| 300 | 293 | u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg]; |
| 301 | 294 | |
| @@ -330,15 +323,15 @@ static int get_bcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, · 28 unmodified lines | ||
| 330 | 323 | return 0; |
| 331 | 324 | } |
| 332 | 325 | |
| 333 | −static inline void reset_bcr(struct kvm_vcpu *vcpu, | |
| 334 | − const struct sys_reg_desc *rd) | |
| 326 | +static void reset_bcr(struct kvm_vcpu *vcpu, | |
| 327 | + const struct sys_reg_desc *rd) | |
| 335 | 328 | { |
| 336 | 329 | vcpu->arch.vcpu_debug_state.dbg_bcr[rd->reg] = rd->val; |
| 337 | 330 | } |
| 338 | 331 | |
| 339 | −static inline bool trap_wvr(struct kvm_vcpu *vcpu, | |
| 340 | − const struct sys_reg_params *p, | |
| 341 | − const struct sys_reg_desc *rd) | |
| 332 | +static bool trap_wvr(struct kvm_vcpu *vcpu, | |
| 333 | + struct sys_reg_params *p, | |
| 334 | + const struct sys_reg_desc *rd) | |
| 342 | 335 | { |
| 343 | 336 | u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg]; |
| 344 | 337 | |
| @@ -373,15 +366,15 @@ static int get_wvr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, · 28 unmodified lines | ||
| 373 | 366 | return 0; |
| 374 | 367 | } |
| 375 | 368 | |
| 376 | −static inline void reset_wvr(struct kvm_vcpu *vcpu, | |
| 377 | − const struct sys_reg_desc *rd) | |
| 369 | +static void reset_wvr(struct kvm_vcpu *vcpu, | |
| 370 | + const struct sys_reg_desc *rd) | |
| 378 | 371 | { |
| 379 | 372 | vcpu->arch.vcpu_debug_state.dbg_wvr[rd->reg] = rd->val; |
| 380 | 373 | } |
| 381 | 374 | |
| 382 | −static inline bool trap_wcr(struct kvm_vcpu *vcpu, | |
| 383 | − const struct sys_reg_params *p, | |
| 384 | − const struct sys_reg_desc *rd) | |
| 375 | +static bool trap_wcr(struct kvm_vcpu *vcpu, | |
| 376 | + struct sys_reg_params *p, | |
| 377 | + const struct sys_reg_desc *rd) | |
| 385 | 378 | { |
| 386 | 379 | u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg]; |
| 387 | 380 | |
| @@ -415,8 +408,8 @@ static int get_wcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, · 27 unmodified lines | ||
| 415 | 408 | return 0; |
| 416 | 409 | } |
| 417 | 410 | |
| 418 | −static inline void reset_wcr(struct kvm_vcpu *vcpu, | |
| 419 | − const struct sys_reg_desc *rd) | |
| 411 | +static void reset_wcr(struct kvm_vcpu *vcpu, | |
| 412 | + const struct sys_reg_desc *rd) | |
| 420 | 413 | { |
| 421 | 414 | vcpu->arch.vcpu_debug_state.dbg_wcr[rd->reg] = rd->val; |
| 422 | 415 | } |
| @@ -687,7 +680,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { · 264 unmodified lines | ||
| 687 | 680 | }; |
| 688 | 681 | |
| 689 | 682 | static bool trap_dbgidr(struct kvm_vcpu *vcpu, |
| 690 | − const struct sys_reg_params *p, | |
| 683 | + struct sys_reg_params *p, | |
| 691 | 684 | const struct sys_reg_desc *r) |
| 692 | 685 | { |
| 693 | 686 | if (p->is_write) { |
| @@ -697,23 +690,23 @@ static bool trap_dbgidr(struct kvm_vcpu *vcpu, · 3 unmodified lines | ||
| 697 | 690 | u64 pfr = read_system_reg(SYS_ID_AA64PFR0_EL1); |
| 698 | 691 | u32 el3 = !!cpuid_feature_extract_field(pfr, ID_AA64PFR0_EL3_SHIFT); |
| 699 | 692 | |
| 700 | − *vcpu_reg(vcpu, p->Rt) = ((((dfr >> ID_AA64DFR0_WRPS_SHIFT) & 0xf) << 28) | | |
| 701 | − (((dfr >> ID_AA64DFR0_BRPS_SHIFT) & 0xf) << 24) | | |
| 702 | − (((dfr >> ID_AA64DFR0_CTX_CMPS_SHIFT) & 0xf) << 20) | | |
| 703 | − (6 << 16) | (el3 << 14) | (el3 << 12)); | |
| 693 | + p->regval = ((((dfr >> ID_AA64DFR0_WRPS_SHIFT) & 0xf) << 28) | | |
| 694 | + (((dfr >> ID_AA64DFR0_BRPS_SHIFT) & 0xf) << 24) | | |
| 695 | + (((dfr >> ID_AA64DFR0_CTX_CMPS_SHIFT) & 0xf) << 20) | |
| 696 | + | (6 << 16) | (el3 << 14) | (el3 << 12)); | |
| 704 | 697 | return true; |
| 705 | 698 | } |
| 706 | 699 | } |
| 707 | 700 | |
| 708 | 701 | static bool trap_debug32(struct kvm_vcpu *vcpu, |
| 709 | − const struct sys_reg_params *p, | |
| 702 | + struct sys_reg_params *p, | |
| 710 | 703 | const struct sys_reg_desc *r) |
| 711 | 704 | { |
| 712 | 705 | if (p->is_write) { |
| 713 | − vcpu_cp14(vcpu, r->reg) = *vcpu_reg(vcpu, p->Rt); | |
| 706 | + vcpu_cp14(vcpu, r->reg) = p->regval; | |
| 714 | 707 | vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; |
| 715 | 708 | } else { |
| 716 | − *vcpu_reg(vcpu, p->Rt) = vcpu_cp14(vcpu, r->reg); | |
| 709 | + p->regval = vcpu_cp14(vcpu, r->reg); | |
| 717 | 710 | } |
| 718 | 711 | |
| 719 | 712 | return true; |
| @@ -730,9 +723,9 @@ static bool trap_debug32(struct kvm_vcpu *vcpu, · 10 unmodified lines | ||
| 730 | 723 | * system is in. |
| 731 | 724 | */ |
| 732 | 725 | |
| 733 | −static inline bool trap_xvr(struct kvm_vcpu *vcpu, | |
| 734 | − const struct sys_reg_params *p, | |
| 735 | − const struct sys_reg_desc *rd) | |
| 726 | +static bool trap_xvr(struct kvm_vcpu *vcpu, | |
| 727 | + struct sys_reg_params *p, | |
| 728 | + const struct sys_reg_desc *rd) | |
| 736 | 729 | { |
| 737 | 730 | u64 *dbg_reg = &vcpu->arch.vcpu_debug_state.dbg_bvr[rd->reg]; |
| 738 | 731 | |
| @@ -740,12 +733,12 @@ static inline bool trap_xvr(struct kvm_vcpu *vcpu, · 1 unmodified line | ||
| 740 | 733 | u64 val = *dbg_reg; |
| 741 | 734 | |
| 742 | 735 | val &= 0xffffffffUL; |
| 743 | − val |= *vcpu_reg(vcpu, p->Rt) << 32; | |
| 736 | + val |= p->regval << 32; | |
| 744 | 737 | *dbg_reg = val; |
| 745 | 738 | |
| 746 | 739 | vcpu->arch.debug_flags |= KVM_ARM64_DEBUG_DIRTY; |
| 747 | 740 | } else { |
| 748 | − *vcpu_reg(vcpu, p->Rt) = *dbg_reg >> 32; | |
| 741 | + p->regval = *dbg_reg >> 32; | |
| 749 | 742 | } |
| 750 | 743 | |
| 751 | 744 | trace_trap_reg(__func__, rd->reg, p->is_write, *dbg_reg); |
| @@ -991,7 +984,7 @@ int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu, struct kvm_run *run) · 239 unmodified lines | ||
| 991 | 984 | * Return 0 if the access has been handled, and -1 if not. |
| 992 | 985 | */ |
| 993 | 986 | static int emulate_cp(struct kvm_vcpu *vcpu, |
| 994 | − const struct sys_reg_params *params, | |
| 987 | + struct sys_reg_params *params, | |
| 995 | 988 | const struct sys_reg_desc *table, |
| 996 | 989 | size_t num) |
| 997 | 990 | { |
| @@ -1062,12 +1055,12 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, · 64 unmodified lines | ||
| 1062 | 1055 | { |
| 1063 | 1056 | struct sys_reg_params params; |
| 1064 | 1057 | u32 hsr = kvm_vcpu_get_hsr(vcpu); |
| 1058 | + int Rt = (hsr >> 5) & 0xf; | |
| 1065 | 1059 | int Rt2 = (hsr >> 10) & 0xf; |
| 1066 | 1060 | |
| 1067 | 1061 | params.is_aarch32 = true; |
| 1068 | 1062 | params.is_32bit = false; |
| 1069 | 1063 | params.CRm = (hsr >> 1) & 0xf; |
| 1070 | − params.Rt = (hsr >> 5) & 0xf; | |
| 1071 | 1064 | params.is_write = ((hsr & 1) == 0); |
| 1072 | 1065 | |
| 1073 | 1066 | params.Op0 = 0; |
| @@ -1076,15 +1069,12 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, · 2 unmodified lines | ||
| 1076 | 1069 | params.CRn = 0; |
| 1077 | 1070 | |
| 1078 | 1071 | /* |
| 1079 | − * Massive hack here. Store Rt2 in the top 32bits so we only | |
| 1080 | − * have one register to deal with. As we use the same trap | |
| 1072 | + * Make a 64-bit value out of Rt and Rt2. As we use the same trap | |
| 1081 | 1073 | * backends between AArch32 and AArch64, we get away with it. |
| 1082 | 1074 | */ |
| 1083 | 1075 | if (params.is_write) { |
| 1084 | − u64 val = *vcpu_reg(vcpu, params.Rt); | |
| 1085 | − val &= 0xffffffff; | |
| 1086 | − val |= *vcpu_reg(vcpu, Rt2) << 32; | |
| 1087 | − *vcpu_reg(vcpu, params.Rt) = val; | |
| 1076 | + params.regval = vcpu_get_reg(vcpu, Rt) & 0xffffffff; | |
| 1077 | + params.regval |= vcpu_get_reg(vcpu, Rt2) << 32; | |
| 1088 | 1078 | } |
| 1089 | 1079 | |
| 1090 | 1080 | if (!emulate_cp(vcpu, ¶ms, target_specific, nr_specific)) |
| @@ -1095,11 +1085,10 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, · 4 unmodified lines | ||
| 1095 | 1085 | unhandled_cp_access(vcpu, ¶ms); |
| 1096 | 1086 | |
| 1097 | 1087 | out: |
| 1098 | − /* Do the opposite hack for the read side */ | |
| 1088 | + /* Split up the value between registers for the read side */ | |
| 1099 | 1089 | if (!params.is_write) { |
| 1100 | − u64 val = *vcpu_reg(vcpu, params.Rt); | |
| 1101 | − val >>= 32; | |
| 1102 | − *vcpu_reg(vcpu, Rt2) = val; | |
| 1090 | + vcpu_set_reg(vcpu, Rt, lower_32_bits(params.regval)); | |
| 1091 | + vcpu_set_reg(vcpu, Rt2, upper_32_bits(params.regval)); | |
| 1103 | 1092 | } |
| 1104 | 1093 | |
| 1105 | 1094 | return 1; |
| @@ -1118,21 +1107,24 @@ static int kvm_handle_cp_32(struct kvm_vcpu *vcpu, · 12 unmodified lines | ||
| 1118 | 1107 | { |
| 1119 | 1108 | struct sys_reg_params params; |
| 1120 | 1109 | u32 hsr = kvm_vcpu_get_hsr(vcpu); |
| 1110 | + int Rt = (hsr >> 5) & 0xf; | |
| 1121 | 1111 | |
| 1122 | 1112 | params.is_aarch32 = true; |
| 1123 | 1113 | params.is_32bit = true; |
| 1124 | 1114 | params.CRm = (hsr >> 1) & 0xf; |
| 1125 | − params.Rt = (hsr >> 5) & 0xf; | |
| 1115 | + params.regval = vcpu_get_reg(vcpu, Rt); | |
| 1126 | 1116 | params.is_write = ((hsr & 1) == 0); |
| 1127 | 1117 | params.CRn = (hsr >> 10) & 0xf; |
| 1128 | 1118 | params.Op0 = 0; |
| 1129 | 1119 | params.Op1 = (hsr >> 14) & 0x7; |
| 1130 | 1120 | params.Op2 = (hsr >> 17) & 0x7; |
| 1131 | 1121 | |
| 1132 | − if (!emulate_cp(vcpu, ¶ms, target_specific, nr_specific)) | |
| 1133 | − return 1; | |
| 1134 | − if (!emulate_cp(vcpu, ¶ms, global, nr_global)) | |
| 1122 | + if (!emulate_cp(vcpu, ¶ms, target_specific, nr_specific) || | |
| 1123 | + !emulate_cp(vcpu, ¶ms, global, nr_global)) { | |
| 1124 | + if (!params.is_write) | |
| 1125 | + vcpu_set_reg(vcpu, Rt, params.regval); | |
| 1135 | 1126 | return 1; |
| 1127 | + } | |
| 1136 | 1128 | |
| 1137 | 1129 | unhandled_cp_access(vcpu, ¶ms); |
| 1138 | 1130 | return 1; |
| @@ -1175,7 +1167,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu, struct kvm_run *run) · 36 unmodified lines | ||
| 1175 | 1167 | } |
| 1176 | 1168 | |
| 1177 | 1169 | static int emulate_sys_reg(struct kvm_vcpu *vcpu, |
| 1178 | − const struct sys_reg_params *params) | |
| 1170 | + struct sys_reg_params *params) | |
| 1179 | 1171 | { |
| 1180 | 1172 | size_t num; |
| 1181 | 1173 | const struct sys_reg_desc *table, *r; |
| @@ -1230,6 +1222,8 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run) · 48 unmodified lines | ||
| 1230 | 1222 | { |
| 1231 | 1223 | struct sys_reg_params params; |
| 1232 | 1224 | unsigned long esr = kvm_vcpu_get_hsr(vcpu); |
| 1225 | + int Rt = (esr >> 5) & 0x1f; | |
| 1226 | + int ret; | |
| 1233 | 1227 | |
| 1234 | 1228 | trace_kvm_handle_sys_reg(esr); |
| 1235 | 1229 | |
| @@ -1240,10 +1234,14 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run) · 4 unmodified lines | ||
| 1240 | 1234 | params.CRn = (esr >> 10) & 0xf; |
| 1241 | 1235 | params.CRm = (esr >> 1) & 0xf; |
| 1242 | 1236 | params.Op2 = (esr >> 17) & 0x7; |
| 1243 | − params.Rt = (esr >> 5) & 0x1f; | |
| 1237 | + params.regval = vcpu_get_reg(vcpu, Rt); | |
| 1244 | 1238 | params.is_write = !(esr & 1); |
| 1245 | 1239 | |
| 1246 | − return emulate_sys_reg(vcpu, ¶ms); | |
| 1240 | + ret = emulate_sys_reg(vcpu, ¶ms); | |
| 1241 | + | |
| 1242 | + if (!params.is_write) | |
| 1243 | + vcpu_set_reg(vcpu, Rt, params.regval); | |
| 1244 | + return ret; | |
| 1247 | 1245 | } |
| 1248 | 1246 | |
| 1249 | 1247 | /****************************************************************************** |
| @@ -28,7 +28,7 @@ struct sys_reg_params { · 27 unmodified lines | ||
| 28 | 28 | u8 CRn; |
| 29 | 29 | u8 CRm; |
| 30 | 30 | u8 Op2; |
| 31 | − u8 Rt; | |
| 31 | + u64 regval; | |
| 32 | 32 | bool is_write; |
| 33 | 33 | bool is_aarch32; |
| 34 | 34 | bool is_32bit; /* Only valid if is_aarch32 is true */ |
| @@ -44,7 +44,7 @@ struct sys_reg_desc { · 9 unmodified lines | ||
| 44 | 44 | |
| 45 | 45 | /* Trapped access from guest, if non-NULL. */ |
| 46 | 46 | bool (*access)(struct kvm_vcpu *, |
| 47 | − const struct sys_reg_params *, | |
| 47 | + struct sys_reg_params *, | |
| 48 | 48 | const struct sys_reg_desc *); |
| 49 | 49 | |
| 50 | 50 | /* Initialization for vcpu. */ |
| @@ -77,9 +77,9 @@ static inline bool ignore_write(struct kvm_vcpu *vcpu, · 26 unmodified lines | ||
| 77 | 77 | } |
| 78 | 78 | |
| 79 | 79 | static inline bool read_zero(struct kvm_vcpu *vcpu, |
| 80 | − const struct sys_reg_params *p) | |
| 80 | + struct sys_reg_params *p) | |
| 81 | 81 | { |
| 82 | − *vcpu_reg(vcpu, p->Rt) = 0; | |
| 82 | + p->regval = 0; | |
| 83 | 83 | return true; |
| 84 | 84 | } |
| 85 | 85 | |
| @@ -31,13 +31,13 @@ · 30 unmodified lines | ||
| 31 | 31 | #include "sys_regs.h" |
| 32 | 32 | |
| 33 | 33 | static bool access_actlr(struct kvm_vcpu *vcpu, |
| 34 | − const struct sys_reg_params *p, | |
| 34 | + struct sys_reg_params *p, | |
| 35 | 35 | const struct sys_reg_desc *r) |
| 36 | 36 | { |
| 37 | 37 | if (p->is_write) |
| 38 | 38 | return ignore_write(vcpu, p); |
| 39 | 39 | |
| 40 | − *vcpu_reg(vcpu, p->Rt) = vcpu_sys_reg(vcpu, ACTLR_EL1); | |
| 40 | + p->regval = vcpu_sys_reg(vcpu, ACTLR_EL1); | |
| 41 | 41 | return true; |
| 42 | 42 | } |
| 43 | 43 | |
| @@ -1,134 +0,0 @@ | ||
| 1 | −/* | |
| 2 | − * Copyright (C) 2012,2013 - ARM Ltd | |
| 3 | − * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | − * | |
| 5 | − * This program is free software; you can redistribute it and/or modify | |
| 6 | − * it under the terms of the GNU General Public License version 2 as | |
| 7 | − * published by the Free Software Foundation. | |
| 8 | − * | |
| 9 | − * This program is distributed in the hope that it will be useful, | |
| 10 | − * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | − * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | − * GNU General Public License for more details. | |
| 13 | − * | |
| 14 | − * You should have received a copy of the GNU General Public License | |
| 15 | − * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | − */ | |
| 17 | − | |
| 18 | −#include <linux/linkage.h> | |
| 19 | −#include <linux/irqchip/arm-gic.h> | |
| 20 | − | |
| 21 | −#include <asm/assembler.h> | |
| 22 | −#include <asm/memory.h> | |
| 23 | −#include <asm/asm-offsets.h> | |
| 24 | −#include <asm/kvm.h> | |
| 25 | −#include <asm/kvm_asm.h> | |
| 26 | −#include <asm/kvm_arm.h> | |
| 27 | −#include <asm/kvm_mmu.h> | |
| 28 | − | |
| 29 | − .text | |
| 30 | − .pushsection .hyp.text, "ax" | |
| 31 | − | |
| 32 | −/* | |
| 33 | − * Save the VGIC CPU state into memory | |
| 34 | − * x0: Register pointing to VCPU struct | |
| 35 | − * Do not corrupt x1!!! | |
| 36 | − */ | |
| 37 | −ENTRY(__save_vgic_v2_state) | |
| 38 | −__save_vgic_v2_state: | |
| 39 | − /* Get VGIC VCTRL base into x2 */ | |
| 40 | − ldr x2, [x0, #VCPU_KVM] | |
| 41 | − kern_hyp_va x2 | |
| 42 | − ldr x2, [x2, #KVM_VGIC_VCTRL] | |
| 43 | − kern_hyp_va x2 | |
| 44 | − cbz x2, 2f // disabled | |
| 45 | − | |
| 46 | − /* Compute the address of struct vgic_cpu */ | |
| 47 | − add x3, x0, #VCPU_VGIC_CPU | |
| 48 | − | |
| 49 | − /* Save all interesting registers */ | |
| 50 | − ldr w5, [x2, #GICH_VMCR] | |
| 51 | − ldr w6, [x2, #GICH_MISR] | |
| 52 | − ldr w7, [x2, #GICH_EISR0] | |
| 53 | − ldr w8, [x2, #GICH_EISR1] | |
| 54 | − ldr w9, [x2, #GICH_ELRSR0] | |
| 55 | − ldr w10, [x2, #GICH_ELRSR1] | |
| 56 | − ldr w11, [x2, #GICH_APR] | |
| 57 | −CPU_BE( rev w5, w5 ) | |
| 58 | −CPU_BE( rev w6, w6 ) | |
| 59 | −CPU_BE( rev w7, w7 ) | |
| 60 | −CPU_BE( rev w8, w8 ) | |
| 61 | −CPU_BE( rev w9, w9 ) | |
| 62 | −CPU_BE( rev w10, w10 ) | |
| 63 | −CPU_BE( rev w11, w11 ) | |
| 64 | − | |
| 65 | − str w5, [x3, #VGIC_V2_CPU_VMCR] | |
| 66 | − str w6, [x3, #VGIC_V2_CPU_MISR] | |
| 67 | −CPU_LE( str w7, [x3, #VGIC_V2_CPU_EISR] ) | |
| 68 | −CPU_LE( str w8, [x3, #(VGIC_V2_CPU_EISR + 4)] ) | |
| 69 | −CPU_LE( str w9, [x3, #VGIC_V2_CPU_ELRSR] ) | |
| 70 | −CPU_LE( str w10, [x3, #(VGIC_V2_CPU_ELRSR + 4)] ) | |
| 71 | −CPU_BE( str w7, [x3, #(VGIC_V2_CPU_EISR + 4)] ) | |
| 72 | −CPU_BE( str w8, [x3, #VGIC_V2_CPU_EISR] ) | |
| 73 | −CPU_BE( str w9, [x3, #(VGIC_V2_CPU_ELRSR + 4)] ) | |
| 74 | −CPU_BE( str w10, [x3, #VGIC_V2_CPU_ELRSR] ) | |
| 75 | − str w11, [x3, #VGIC_V2_CPU_APR] | |
| 76 | − | |
| 77 | − /* Clear GICH_HCR */ | |
| 78 | − str wzr, [x2, #GICH_HCR] | |
| 79 | − | |
| 80 | − /* Save list registers */ | |
| 81 | − add x2, x2, #GICH_LR0 | |
| 82 | − ldr w4, [x3, #VGIC_CPU_NR_LR] | |
| 83 | − add x3, x3, #VGIC_V2_CPU_LR | |
| 84 | −1: ldr w5, [x2], #4 | |
| 85 | −CPU_BE( rev w5, w5 ) | |
| 86 | − str w5, [x3], #4 | |
| 87 | − sub w4, w4, #1 | |
| 88 | − cbnz w4, 1b | |
| 89 | −2: | |
| 90 | − ret | |
| 91 | −ENDPROC(__save_vgic_v2_state) | |
| 92 | − | |
| 93 | −/* | |
| 94 | − * Restore the VGIC CPU state from memory | |
| 95 | − * x0: Register pointing to VCPU struct | |
| 96 | − */ | |
| 97 | −ENTRY(__restore_vgic_v2_state) | |
| 98 | −__restore_vgic_v2_state: | |
| 99 | − /* Get VGIC VCTRL base into x2 */ | |
| 100 | − ldr x2, [x0, #VCPU_KVM] | |
| 101 | − kern_hyp_va x2 | |
| 102 | − ldr x2, [x2, #KVM_VGIC_VCTRL] | |
| 103 | − kern_hyp_va x2 | |
| 104 | − cbz x2, 2f // disabled | |
| 105 | − | |
| 106 | − /* Compute the address of struct vgic_cpu */ | |
| 107 | − add x3, x0, #VCPU_VGIC_CPU | |
| 108 | − | |
| 109 | − /* We only restore a minimal set of registers */ | |
| 110 | − ldr w4, [x3, #VGIC_V2_CPU_HCR] | |
| 111 | − ldr w5, [x3, #VGIC_V2_CPU_VMCR] | |
| 112 | − ldr w6, [x3, #VGIC_V2_CPU_APR] | |
| 113 | −CPU_BE( rev w4, w4 ) | |
| 114 | −CPU_BE( rev w5, w5 ) | |
| 115 | −CPU_BE( rev w6, w6 ) | |
| 116 | − | |
| 117 | − str w4, [x2, #GICH_HCR] | |
| 118 | − str w5, [x2, #GICH_VMCR] | |
| 119 | − str w6, [x2, #GICH_APR] | |
| 120 | − | |
| 121 | − /* Restore list registers */ | |
| 122 | − add x2, x2, #GICH_LR0 | |
| 123 | − ldr w4, [x3, #VGIC_CPU_NR_LR] | |
| 124 | − add x3, x3, #VGIC_V2_CPU_LR | |
| 125 | −1: ldr w5, [x3], #4 | |
| 126 | −CPU_BE( rev w5, w5 ) | |
| 127 | − str w5, [x2], #4 | |
| 128 | − sub w4, w4, #1 | |
| 129 | − cbnz w4, 1b | |
| 130 | −2: | |
| 131 | − ret | |
| 132 | −ENDPROC(__restore_vgic_v2_state) | |
| 133 | − | |
| 134 | − .popsection | |
| @@ -1,269 +0,0 @@ | ||
| 1 | −/* | |
| 2 | − * Copyright (C) 2012,2013 - ARM Ltd | |
| 3 | − * Author: Marc Zyngier <marc.zyngier@arm.com> | |
| 4 | − * | |
| 5 | − * This program is free software; you can redistribute it and/or modify | |
| 6 | − * it under the terms of the GNU General Public License version 2 as | |
| 7 | − * published by the Free Software Foundation. | |
| 8 | − * | |
| 9 | − * This program is distributed in the hope that it will be useful, | |
| 10 | − * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 11 | − * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 12 | − * GNU General Public License for more details. | |
| 13 | − * | |
| 14 | − * You should have received a copy of the GNU General Public License | |
| 15 | − * along with this program. If not, see <http://www.gnu.org/licenses/>. | |
| 16 | − */ | |
| 17 | − | |
| 18 | −#include <linux/linkage.h> | |
| 19 | −#include <linux/irqchip/arm-gic-v3.h> | |
| 20 | − | |
| 21 | −#include <asm/assembler.h> | |
| 22 | −#include <asm/memory.h> | |
| 23 | −#include <asm/asm-offsets.h> | |
| 24 | −#include <asm/kvm.h> | |
| 25 | −#include <asm/kvm_asm.h> | |
| 26 | −#include <asm/kvm_arm.h> | |
| 27 | − | |
| 28 | − .text | |
| 29 | − .pushsection .hyp.text, "ax" | |
| 30 | − | |
| 31 | −/* | |
| 32 | − * We store LRs in reverse order to let the CPU deal with streaming | |
| 33 | − * access. Use this macro to make it look saner... | |
| 34 | − */ | |
| 35 | −#define LR_OFFSET(n) (VGIC_V3_CPU_LR + (15 - n) * 8) | |
| 36 | − | |
| 37 | −/* | |
| 38 | − * Save the VGIC CPU state into memory | |
| 39 | − * x0: Register pointing to VCPU struct | |
| 40 | − * Do not corrupt x1!!! | |
| 41 | − */ | |
| 42 | −.macro save_vgic_v3_state | |
| 43 | − // Compute the address of struct vgic_cpu | |
| 44 | − add x3, x0, #VCPU_VGIC_CPU | |
| 45 | − | |
| 46 | − // Make sure stores to the GIC via the memory mapped interface | |
| 47 | − // are now visible to the system register interface | |
| 48 | − dsb st | |
| 49 | − | |
| 50 | − // Save all interesting registers | |
| 51 | − mrs_s x5, ICH_VMCR_EL2 | |
| 52 | − mrs_s x6, ICH_MISR_EL2 | |
| 53 | − mrs_s x7, ICH_EISR_EL2 | |
| 54 | − mrs_s x8, ICH_ELSR_EL2 | |
| 55 | − | |
| 56 | − str w5, [x3, #VGIC_V3_CPU_VMCR] | |
| 57 | − str w6, [x3, #VGIC_V3_CPU_MISR] | |
| 58 | − str w7, [x3, #VGIC_V3_CPU_EISR] | |
| 59 | − str w8, [x3, #VGIC_V3_CPU_ELRSR] | |
| 60 | − | |
| 61 | − msr_s ICH_HCR_EL2, xzr | |
| 62 | − | |
| 63 | − mrs_s x21, ICH_VTR_EL2 | |
| 64 | − mvn w22, w21 | |
| 65 | − ubfiz w23, w22, 2, 4 // w23 = (15 - ListRegs) * 4 | |
| 66 | − | |
| 67 | − adr x24, 1f | |
| 68 | − add x24, x24, x23 | |
| 69 | − br x24 | |
| 70 | − | |
| 71 | −1: | |
| 72 | − mrs_s x20, ICH_LR15_EL2 | |
| 73 | − mrs_s x19, ICH_LR14_EL2 | |
| 74 | − mrs_s x18, ICH_LR13_EL2 | |
| 75 | − mrs_s x17, ICH_LR12_EL2 | |
| 76 | − mrs_s x16, ICH_LR11_EL2 | |
| 77 | − mrs_s x15, ICH_LR10_EL2 | |
| 78 | − mrs_s x14, ICH_LR9_EL2 | |
| 79 | − mrs_s x13, ICH_LR8_EL2 | |
| 80 | − mrs_s x12, ICH_LR7_EL2 | |
| 81 | − mrs_s x11, ICH_LR6_EL2 | |
| 82 | − mrs_s x10, ICH_LR5_EL2 | |
| 83 | − mrs_s x9, ICH_LR4_EL2 | |
| 84 | − mrs_s x8, ICH_LR3_EL2 | |
| 85 | − mrs_s x7, ICH_LR2_EL2 | |
| 86 | − mrs_s x6, ICH_LR1_EL2 | |
| 87 | − mrs_s x5, ICH_LR0_EL2 | |
| 88 | − | |
| 89 | − adr x24, 1f | |
| 90 | − add x24, x24, x23 | |
| 91 | − br x24 | |
| 92 | − | |
| 93 | −1: | |
| 94 | − str x20, [x3, #LR_OFFSET(15)] | |
| 95 | − str x19, [x3, #LR_OFFSET(14)] | |
| 96 | − str x18, [x3, #LR_OFFSET(13)] | |
| 97 | − str x17, [x3, #LR_OFFSET(12)] | |
| 98 | − str x16, [x3, #LR_OFFSET(11)] | |
| 99 | − str x15, [x3, #LR_OFFSET(10)] | |
| 100 | − str x14, [x3, #LR_OFFSET(9)] | |
| 101 | − str x13, [x3, #LR_OFFSET(8)] | |
| 102 | − str x12, [x3, #LR_OFFSET(7)] | |
| 103 | − str x11, [x3, #LR_OFFSET(6)] | |
| 104 | − str x10, [x3, #LR_OFFSET(5)] | |
| 105 | − str x9, [x3, #LR_OFFSET(4)] | |
| 106 | − str x8, [x3, #LR_OFFSET(3)] | |
| 107 | − str x7, [x3, #LR_OFFSET(2)] | |
| 108 | − str x6, [x3, #LR_OFFSET(1)] | |
| 109 | − str x5, [x3, #LR_OFFSET(0)] | |
| 110 | − | |
| 111 | − tbnz w21, #29, 6f // 6 bits | |
| 112 | − tbz w21, #30, 5f // 5 bits | |
| 113 | − // 7 bits | |
| 114 | − mrs_s x20, ICH_AP0R3_EL2 | |
| 115 | − str w20, [x3, #(VGIC_V3_CPU_AP0R + 3*4)] | |
| 116 | − mrs_s x19, ICH_AP0R2_EL2 | |
| 117 | − str w19, [x3, #(VGIC_V3_CPU_AP0R + 2*4)] | |
| 118 | −6: mrs_s x18, ICH_AP0R1_EL2 | |
| 119 | − str w18, [x3, #(VGIC_V3_CPU_AP0R + 1*4)] | |
| 120 | −5: mrs_s x17, ICH_AP0R0_EL2 | |
| 121 | − str w17, [x3, #VGIC_V3_CPU_AP0R] | |
| 122 | − | |
| 123 | − tbnz w21, #29, 6f // 6 bits | |
| 124 | − tbz w21, #30, 5f // 5 bits | |
| 125 | − // 7 bits | |
| 126 | − mrs_s x20, ICH_AP1R3_EL2 | |
| 127 | − str w20, [x3, #(VGIC_V3_CPU_AP1R + 3*4)] | |
| 128 | − mrs_s x19, ICH_AP1R2_EL2 | |
| 129 | − str w19, [x3, #(VGIC_V3_CPU_AP1R + 2*4)] | |
| 130 | −6: mrs_s x18, ICH_AP1R1_EL2 | |
| 131 | − str w18, [x3, #(VGIC_V3_CPU_AP1R + 1*4)] | |
| 132 | −5: mrs_s x17, ICH_AP1R0_EL2 | |
| 133 | − str w17, [x3, #VGIC_V3_CPU_AP1R] | |
| 134 | − | |
| 135 | − // Restore SRE_EL1 access and re-enable SRE at EL1. | |
| 136 | − mrs_s x5, ICC_SRE_EL2 | |
| 137 | − orr x5, x5, #ICC_SRE_EL2_ENABLE | |
| 138 | − msr_s ICC_SRE_EL2, x5 | |
| 139 | − isb | |
| 140 | − mov x5, #1 | |
| 141 | − msr_s ICC_SRE_EL1, x5 | |
| 142 | −.endm | |
| 143 | − | |
| 144 | −/* | |
| 145 | − * Restore the VGIC CPU state from memory | |
| 146 | − * x0: Register pointing to VCPU struct | |
| 147 | − */ | |
| 148 | −.macro restore_vgic_v3_state | |
| 149 | − // Compute the address of struct vgic_cpu | |
| 150 | − add x3, x0, #VCPU_VGIC_CPU | |
| 151 | − | |
| 152 | − // Restore all interesting registers | |
| 153 | − ldr w4, [x3, #VGIC_V3_CPU_HCR] | |
| 154 | − ldr w5, [x3, #VGIC_V3_CPU_VMCR] | |
| 155 | − ldr w25, [x3, #VGIC_V3_CPU_SRE] | |
| 156 | − | |
| 157 | − msr_s ICC_SRE_EL1, x25 | |
| 158 | − | |
| 159 | − // make sure SRE is valid before writing the other registers | |
| 160 | − isb | |
| 161 | − | |
| 162 | − msr_s ICH_HCR_EL2, x4 | |
| 163 | − msr_s ICH_VMCR_EL2, x5 | |
| 164 | − | |
| 165 | − mrs_s x21, ICH_VTR_EL2 | |
| 166 | − | |
| 167 | − tbnz w21, #29, 6f // 6 bits | |
| 168 | − tbz w21, #30, 5f // 5 bits | |
| 169 | − // 7 bits | |
| 170 | − ldr w20, [x3, #(VGIC_V3_CPU_AP1R + 3*4)] | |
| 171 | − msr_s ICH_AP1R3_EL2, x20 | |
| 172 | − ldr w19, [x3, #(VGIC_V3_CPU_AP1R + 2*4)] | |
| 173 | − msr_s ICH_AP1R2_EL2, x19 | |
| 174 | −6: ldr w18, [x3, #(VGIC_V3_CPU_AP1R + 1*4)] | |
| 175 | − msr_s ICH_AP1R1_EL2, x18 | |
| 176 | −5: ldr w17, [x3, #VGIC_V3_CPU_AP1R] | |
| 177 | − msr_s ICH_AP1R0_EL2, x17 | |
| 178 | − | |
| 179 | − tbnz w21, #29, 6f // 6 bits | |
| 180 | − tbz w21, #30, 5f // 5 bits | |
| 181 | − // 7 bits | |
| 182 | − ldr w20, [x3, #(VGIC_V3_CPU_AP0R + 3*4)] | |
| 183 | − msr_s ICH_AP0R3_EL2, x20 | |
| 184 | − ldr w19, [x3, #(VGIC_V3_CPU_AP0R + 2*4)] | |
| 185 | − msr_s ICH_AP0R2_EL2, x19 | |
| 186 | −6: ldr w18, [x3, #(VGIC_V3_CPU_AP0R + 1*4)] | |
| 187 | − msr_s ICH_AP0R1_EL2, x18 | |
| 188 | −5: ldr w17, [x3, #VGIC_V3_CPU_AP0R] | |
| 189 | − msr_s ICH_AP0R0_EL2, x17 | |
| 190 | − | |
| 191 | − and w22, w21, #0xf | |
| 192 | − mvn w22, w21 | |
| 193 | − ubfiz w23, w22, 2, 4 // w23 = (15 - ListRegs) * 4 | |
| 194 | − | |
| 195 | − adr x24, 1f | |
| 196 | − add x24, x24, x23 | |
| 197 | − br x24 | |
| 198 | − | |
| 199 | −1: | |
| 200 | − ldr x20, [x3, #LR_OFFSET(15)] | |
| 201 | − ldr x19, [x3, #LR_OFFSET(14)] | |
| 202 | − ldr x18, [x3, #LR_OFFSET(13)] | |
| 203 | − ldr x17, [x3, #LR_OFFSET(12)] | |
| 204 | − ldr x16, [x3, #LR_OFFSET(11)] | |
| 205 | − ldr x15, [x3, #LR_OFFSET(10)] | |
| 206 | − ldr x14, [x3, #LR_OFFSET(9)] | |
| 207 | − ldr x13, [x3, #LR_OFFSET(8)] | |
| 208 | − ldr x12, [x3, #LR_OFFSET(7)] | |
| 209 | − ldr x11, [x3, #LR_OFFSET(6)] | |
| 210 | − ldr x10, [x3, #LR_OFFSET(5)] | |
| 211 | − ldr x9, [x3, #LR_OFFSET(4)] | |
| 212 | − ldr x8, [x3, #LR_OFFSET(3)] | |
| 213 | − ldr x7, [x3, #LR_OFFSET(2)] | |
| 214 | − ldr x6, [x3, #LR_OFFSET(1)] | |
| 215 | − ldr x5, [x3, #LR_OFFSET(0)] | |
| 216 | − | |
| 217 | − adr x24, 1f | |
| 218 | − add x24, x24, x23 | |
| 219 | − br x24 | |
| 220 | − | |
| 221 | −1: | |
| 222 | − msr_s ICH_LR15_EL2, x20 | |
| 223 | − msr_s ICH_LR14_EL2, x19 | |
| 224 | − msr_s ICH_LR13_EL2, x18 | |
| 225 | − msr_s ICH_LR12_EL2, x17 | |
| 226 | − msr_s ICH_LR11_EL2, x16 | |
| 227 | − msr_s ICH_LR10_EL2, x15 | |
| 228 | − msr_s ICH_LR9_EL2, x14 | |
| 229 | − msr_s ICH_LR8_EL2, x13 | |
| 230 | − msr_s ICH_LR7_EL2, x12 | |
| 231 | − msr_s ICH_LR6_EL2, x11 | |
| 232 | − msr_s ICH_LR5_EL2, x10 | |
| 233 | − msr_s ICH_LR4_EL2, x9 | |
| 234 | − msr_s ICH_LR3_EL2, x8 | |
| 235 | − msr_s ICH_LR2_EL2, x7 | |
| 236 | − msr_s ICH_LR1_EL2, x6 | |
| 237 | − msr_s ICH_LR0_EL2, x5 | |
| 238 | − | |
| 239 | − // Ensure that the above will have reached the | |
| 240 | − // (re)distributors. This ensure the guest will read | |
| 241 | − // the correct values from the memory-mapped interface. | |
| 242 | − isb | |
| 243 | − dsb sy | |
| 244 | − | |
| 245 | − // Prevent the guest from touching the GIC system registers | |
| 246 | − // if SRE isn't enabled for GICv3 emulation | |
| 247 | − cbnz x25, 1f | |
| 248 | − mrs_s x5, ICC_SRE_EL2 | |
| 249 | − and x5, x5, #~ICC_SRE_EL2_ENABLE | |
| 250 | − msr_s ICC_SRE_EL2, x5 | |
| 251 | −1: | |
| 252 | −.endm | |
| 253 | − | |
| 254 | −ENTRY(__save_vgic_v3_state) | |
| 255 | − save_vgic_v3_state | |
| 256 | − ret | |
| 257 | −ENDPROC(__save_vgic_v3_state) | |
| 258 | − | |
| 259 | −ENTRY(__restore_vgic_v3_state) | |
| 260 | − restore_vgic_v3_state | |
| 261 | − ret | |
| 262 | −ENDPROC(__restore_vgic_v3_state) | |
| 263 | − | |
| 264 | −ENTRY(__vgic_v3_get_ich_vtr_el2) | |
| 265 | − mrs_s x0, ICH_VTR_EL2 | |
| 266 | − ret | |
| 267 | −ENDPROC(__vgic_v3_get_ich_vtr_el2) | |
| 268 | − | |
| 269 | − .popsection | |
| @@ -76,13 +76,28 @@ static void flush_context(unsigned int cpu) · 75 unmodified lines | ||
| 76 | 76 | __flush_icache_all(); |
| 77 | 77 | } |
| 78 | 78 | |
| 79 | −static int is_reserved_asid(u64 asid) | |
| 79 | +static bool check_update_reserved_asid(u64 asid, u64 newasid) | |
| 80 | 80 | { |
| 81 | 81 | int cpu; |
| 82 | − for_each_possible_cpu(cpu) | |
| 83 | − if (per_cpu(reserved_asids, cpu) == asid) | |
| 84 | − return 1; | |
| 85 | − return 0; | |
| 82 | + bool hit = false; | |
| 83 | + | |
| 84 | + /* | |
| 85 | + * Iterate over the set of reserved ASIDs looking for a match. | |
| 86 | + * If we find one, then we can update our mm to use newasid | |
| 87 | + * (i.e. the same ASID in the current generation) but we can't | |
| 88 | + * exit the loop early, since we need to ensure that all copies | |
| 89 | + * of the old ASID are updated to reflect the mm. Failure to do | |
| 90 | + * so could result in us missing the reserved ASID in a future | |
| 91 | + * generation. | |
| 92 | + */ | |
| 93 | + for_each_possible_cpu(cpu) { | |
| 94 | + if (per_cpu(reserved_asids, cpu) == asid) { | |
| 95 | + hit = true; | |
| 96 | + per_cpu(reserved_asids, cpu) = newasid; | |
| 97 | + } | |
| 98 | + } | |
| 99 | + | |
| 100 | + return hit; | |
| 86 | 101 | } |
| 87 | 102 | |
| 88 | 103 | static u64 new_context(struct mm_struct *mm, unsigned int cpu) |
| @@ -92,12 +107,14 @@ static u64 new_context(struct mm_struct *mm, unsigned int cpu) · 3 unmodified lines | ||
| 92 | 107 | u64 generation = atomic64_read(&asid_generation); |
| 93 | 108 | |
| 94 | 109 | if (asid != 0) { |
| 110 | + u64 newasid = generation | (asid & ~ASID_MASK); | |
| 111 | + | |
| 95 | 112 | /* |
| 96 | 113 | * If our current ASID was active during a rollover, we |
| 97 | 114 | * can continue to use it and this was just a false alarm. |
| 98 | 115 | */ |
| 99 | − if (is_reserved_asid(asid)) | |
| 100 | − return generation | (asid & ~ASID_MASK); | |
| 116 | + if (check_update_reserved_asid(asid, newasid)) | |
| 117 | + return newasid; | |
| 101 | 118 | |
| 102 | 119 | /* |
| 103 | 120 | * We had a valid ASID in a previous life, so try to re-use |
| @@ -105,7 +122,7 @@ static u64 new_context(struct mm_struct *mm, unsigned int cpu) · 1 unmodified line | ||
| 105 | 122 | */ |
| 106 | 123 | asid &= ~ASID_MASK; |
| 107 | 124 | if (!__test_and_set_bit(asid, asid_map)) |
| 108 | − goto bump_gen; | |
| 125 | + return newasid; | |
| 109 | 126 | } |
| 110 | 127 | |
| 111 | 128 | /* |
| @@ -129,10 +146,7 @@ static u64 new_context(struct mm_struct *mm, unsigned int cpu) · 17 unmodified lines | ||
| 129 | 146 | set_asid: |
| 130 | 147 | __set_bit(asid, asid_map); |
| 131 | 148 | cur_idx = asid; |
| 132 | − | |
| 133 | −bump_gen: | |
| 134 | − asid |= generation; | |
| 135 | − return asid; | |
| 149 | + return asid | generation; | |
| 136 | 150 | } |
| 137 | 151 | |
| 138 | 152 | void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) |
| @@ -393,16 +393,16 @@ static struct fault_info { · 392 unmodified lines | ||
| 393 | 393 | { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 1 translation fault" }, |
| 394 | 394 | { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 2 translation fault" }, |
| 395 | 395 | { do_page_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, |
| 396 | − { do_bad, SIGBUS, 0, "reserved access flag fault" }, | |
| 396 | + { do_bad, SIGBUS, 0, "unknown 8" }, | |
| 397 | 397 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, |
| 398 | 398 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, |
| 399 | 399 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, |
| 400 | − { do_bad, SIGBUS, 0, "reserved permission fault" }, | |
| 400 | + { do_bad, SIGBUS, 0, "unknown 12" }, | |
| 401 | 401 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, |
| 402 | 402 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, |
| 403 | 403 | { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, |
| 404 | 404 | { do_bad, SIGBUS, 0, "synchronous external abort" }, |
| 405 | − { do_bad, SIGBUS, 0, "asynchronous external abort" }, | |
| 405 | + { do_bad, SIGBUS, 0, "unknown 17" }, | |
| 406 | 406 | { do_bad, SIGBUS, 0, "unknown 18" }, |
| 407 | 407 | { do_bad, SIGBUS, 0, "unknown 19" }, |
| 408 | 408 | { do_bad, SIGBUS, 0, "synchronous abort (translation table walk)" }, |
| @@ -410,16 +410,16 @@ static struct fault_info { · 1 unmodified line | ||
| 410 | 410 | { do_bad, SIGBUS, 0, "synchronous abort (translation table walk)" }, |
| 411 | 411 | { do_bad, SIGBUS, 0, "synchronous abort (translation table walk)" }, |
| 412 | 412 | { do_bad, SIGBUS, 0, "synchronous parity error" }, |
| 413 | − { do_bad, SIGBUS, 0, "asynchronous parity error" }, | |
| 413 | + { do_bad, SIGBUS, 0, "unknown 25" }, | |
| 414 | 414 | { do_bad, SIGBUS, 0, "unknown 26" }, |
| 415 | 415 | { do_bad, SIGBUS, 0, "unknown 27" }, |
| 416 | − { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk" }, | |
| 417 | − { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk" }, | |
| 418 | − { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk" }, | |
| 419 | − { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk" }, | |
| 416 | + { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" }, | |
| 417 | + { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" }, | |
| 418 | + { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" }, | |
| 419 | + { do_bad, SIGBUS, 0, "synchronous parity error (translation table walk)" }, | |
| 420 | 420 | { do_bad, SIGBUS, 0, "unknown 32" }, |
| 421 | 421 | { do_bad, SIGBUS, BUS_ADRALN, "alignment fault" }, |
| 422 | − { do_bad, SIGBUS, 0, "debug event" }, | |
| 422 | + { do_bad, SIGBUS, 0, "unknown 34" }, | |
| 423 | 423 | { do_bad, SIGBUS, 0, "unknown 35" }, |
| 424 | 424 | { do_bad, SIGBUS, 0, "unknown 36" }, |
| 425 | 425 | { do_bad, SIGBUS, 0, "unknown 37" }, |
| @@ -433,21 +433,21 @@ static struct fault_info { · 7 unmodified lines | ||
| 433 | 433 | { do_bad, SIGBUS, 0, "unknown 45" }, |
| 434 | 434 | { do_bad, SIGBUS, 0, "unknown 46" }, |
| 435 | 435 | { do_bad, SIGBUS, 0, "unknown 47" }, |
| 436 | − { do_bad, SIGBUS, 0, "unknown 48" }, | |
| 436 | + { do_bad, SIGBUS, 0, "TLB conflict abort" }, | |
| 437 | 437 | { do_bad, SIGBUS, 0, "unknown 49" }, |
| 438 | 438 | { do_bad, SIGBUS, 0, "unknown 50" }, |
| 439 | 439 | { do_bad, SIGBUS, 0, "unknown 51" }, |
| 440 | 440 | { do_bad, SIGBUS, 0, "implementation fault (lockdown abort)" }, |
| 441 | − { do_bad, SIGBUS, 0, "unknown 53" }, | |
| 441 | + { do_bad, SIGBUS, 0, "implementation fault (unsupported exclusive)" }, | |
| 442 | 442 | { do_bad, SIGBUS, 0, "unknown 54" }, |
| 443 | 443 | { do_bad, SIGBUS, 0, "unknown 55" }, |
| 444 | 444 | { do_bad, SIGBUS, 0, "unknown 56" }, |
| 445 | 445 | { do_bad, SIGBUS, 0, "unknown 57" }, |
| 446 | − { do_bad, SIGBUS, 0, "implementation fault (coprocessor abort)" }, | |
| 446 | + { do_bad, SIGBUS, 0, "unknown 58" }, | |
| 447 | 447 | { do_bad, SIGBUS, 0, "unknown 59" }, |
| 448 | 448 | { do_bad, SIGBUS, 0, "unknown 60" }, |
| 449 | − { do_bad, SIGBUS, 0, "unknown 61" }, | |
| 450 | − { do_bad, SIGBUS, 0, "unknown 62" }, | |
| 449 | + { do_bad, SIGBUS, 0, "section domain fault" }, | |
| 450 | + { do_bad, SIGBUS, 0, "page domain fault" }, | |
| 451 | 451 | { do_bad, SIGBUS, 0, "unknown 63" }, |
| 452 | 452 | }; |
| 453 | 453 | |
| @@ -64,8 +64,12 @@ EXPORT_SYMBOL(phys_mem_access_prot); · 63 unmodified lines | ||
| 64 | 64 | |
| 65 | 65 | static void __init *early_alloc(unsigned long sz) |
| 66 | 66 | { |
| 67 | − void *ptr = __va(memblock_alloc(sz, sz)); | |
| 68 | − BUG_ON(!ptr); | |
| 67 | + phys_addr_t phys; | |
| 68 | + void *ptr; | |
| 69 | + | |
| 70 | + phys = memblock_alloc(sz, sz); | |
| 71 | + BUG_ON(!phys); | |
| 72 | + ptr = __va(phys); | |
| 69 | 73 | memset(ptr, 0, sz); |
| 70 | 74 | return ptr; |
| 71 | 75 | } |
| @@ -81,55 +85,19 @@ static void split_pmd(pmd_t *pmd, pte_t *pte) · 9 unmodified lines | ||
| 81 | 85 | do { |
| 82 | 86 | /* |
| 83 | 87 | * Need to have the least restrictive permissions available |
| 84 | − * permissions will be fixed up later. Default the new page | |
| 85 | − * range as contiguous ptes. | |
| 88 | + * permissions will be fixed up later | |
| 86 | 89 | */ |
| 87 | − set_pte(pte, pfn_pte(pfn, PAGE_KERNEL_EXEC_CONT)); | |
| 90 | + set_pte(pte, pfn_pte(pfn, PAGE_KERNEL_EXEC)); | |
| 88 | 91 | pfn++; |
| 89 | 92 | } while (pte++, i++, i < PTRS_PER_PTE); |
| 90 | 93 | } |
| 91 | 94 | |
| 92 | −/* | |
| 93 | − * Given a PTE with the CONT bit set, determine where the CONT range | |
| 94 | − * starts, and clear the entire range of PTE CONT bits. | |
| 95 | − */ | |
| 96 | −static void clear_cont_pte_range(pte_t *pte, unsigned long addr) | |
| 97 | −{ | |
| 98 | − int i; | |
| 99 | − | |
| 100 | − pte -= CONT_RANGE_OFFSET(addr); | |
| 101 | − for (i = 0; i < CONT_PTES; i++) { | |
| 102 | − set_pte(pte, pte_mknoncont(*pte)); | |
| 103 | − pte++; | |
| 104 | − } | |
| 105 | − flush_tlb_all(); | |
| 106 | −} | |
| 107 | − | |
| 108 | −/* | |
| 109 | − * Given a range of PTEs set the pfn and provided page protection flags | |
| 110 | − */ | |
| 111 | −static void __populate_init_pte(pte_t *pte, unsigned long addr, | |
| 112 | − unsigned long end, phys_addr_t phys, | |
| 113 | − pgprot_t prot) | |
| 114 | −{ | |
| 115 | − unsigned long pfn = __phys_to_pfn(phys); | |
| 116 | − | |
| 117 | − do { | |
| 118 | − /* clear all the bits except the pfn, then apply the prot */ | |
| 119 | − set_pte(pte, pfn_pte(pfn, prot)); | |
| 120 | − pte++; | |
| 121 | − pfn++; | |
| 122 | − addr += PAGE_SIZE; | |
| 123 | − } while (addr != end); | |
| 124 | −} | |
| 125 | − | |
| 126 | 95 | static void alloc_init_pte(pmd_t *pmd, unsigned long addr, |
| 127 | − unsigned long end, phys_addr_t phys, | |
| 96 | + unsigned long end, unsigned long pfn, | |
| 128 | 97 | pgprot_t prot, |
| 129 | 98 | void *(*alloc)(unsigned long size)) |
| 130 | 99 | { |
| 131 | 100 | pte_t *pte; |
| 132 | − unsigned long next; | |
| 133 | 101 | |
| 134 | 102 | if (pmd_none(*pmd) || pmd_sect(*pmd)) { |
| 135 | 103 | pte = alloc(PTRS_PER_PTE * sizeof(pte_t)); |
| @@ -142,27 +110,9 @@ static void alloc_init_pte(pmd_t *pmd, unsigned long addr, · 6 unmodified lines | ||
| 142 | 110 | |
| 143 | 111 | pte = pte_offset_kernel(pmd, addr); |
| 144 | 112 | do { |
| 145 | − next = min(end, (addr + CONT_SIZE) & CONT_MASK); | |
| 146 | − if (((addr | next | phys) & ~CONT_MASK) == 0) { | |
| 147 | − /* a block of CONT_PTES */ | |
| 148 | − __populate_init_pte(pte, addr, next, phys, | |
| 149 | − __pgprot(pgprot_val(prot) | PTE_CONT)); | |
| 150 | − } else { | |
| 151 | − /* | |
| 152 | − * If the range being split is already inside of a | |
| 153 | − * contiguous range but this PTE isn't going to be | |
| 154 | − * contiguous, then we want to unmark the adjacent | |
| 155 | − * ranges, then update the portion of the range we | |
| 156 | − * are interrested in. | |
| 157 | − */ | |
| 158 | − clear_cont_pte_range(pte, addr); | |
| 159 | − __populate_init_pte(pte, addr, next, phys, prot); | |
| 160 | − } | |
| 161 | − | |
| 162 | − pte += (next - addr) >> PAGE_SHIFT; | |
| 163 | − phys += next - addr; | |
| 164 | − addr = next; | |
| 165 | − } while (addr != end); | |
| 113 | + set_pte(pte, pfn_pte(pfn, prot)); | |
| 114 | + pfn++; | |
| 115 | + } while (pte++, addr += PAGE_SIZE, addr != end); | |
| 166 | 116 | } |
| 167 | 117 | |
| 168 | 118 | static void split_pud(pud_t *old_pud, pmd_t *pmd) |
| @@ -223,7 +173,8 @@ static void alloc_init_pmd(struct mm_struct *mm, pud_t *pud, · 54 unmodified lines | ||
| 223 | 173 | } |
| 224 | 174 | } |
| 225 | 175 | } else { |
| 226 | − alloc_init_pte(pmd, addr, next, phys, prot, alloc); | |
| 176 | + alloc_init_pte(pmd, addr, next, __phys_to_pfn(phys), | |
| 177 | + prot, alloc); | |
| 227 | 178 | } |
| 228 | 179 | phys += next - addr; |
| 229 | 180 | } while (pmd++, addr = next, addr != end); |
| @@ -139,6 +139,12 @@ static inline int epilogue_offset(const struct jit_ctx *ctx) · 138 unmodified lines | ||
| 139 | 139 | /* Stack must be multiples of 16B */ |
| 140 | 140 | #define STACK_ALIGN(sz) (((sz) + 15) & ~15) |
| 141 | 141 | |
| 142 | +#define _STACK_SIZE \ | |
| 143 | + (MAX_BPF_STACK \ | |
| 144 | + + 4 /* extra for skb_copy_bits buffer */) | |
| 145 | + | |
| 146 | +#define STACK_SIZE STACK_ALIGN(_STACK_SIZE) | |
| 147 | + | |
| 142 | 148 | static void build_prologue(struct jit_ctx *ctx) |
| 143 | 149 | { |
| 144 | 150 | const u8 r6 = bpf2a64[BPF_REG_6]; |
| @@ -150,10 +156,6 @@ static void build_prologue(struct jit_ctx *ctx) · 5 unmodified lines | ||
| 150 | 156 | const u8 rx = bpf2a64[BPF_REG_X]; |
| 151 | 157 | const u8 tmp1 = bpf2a64[TMP_REG_1]; |
| 152 | 158 | const u8 tmp2 = bpf2a64[TMP_REG_2]; |
| 153 | − int stack_size = MAX_BPF_STACK; | |
| 154 | − | |
| 155 | − stack_size += 4; /* extra for skb_copy_bits buffer */ | |
| 156 | − stack_size = STACK_ALIGN(stack_size); | |
| 157 | 159 | |
| 158 | 160 | /* |
| 159 | 161 | * BPF prog stack layout |
| @@ -165,12 +167,13 @@ static void build_prologue(struct jit_ctx *ctx) · 5 unmodified lines | ||
| 165 | 167 | * | ... | callee saved registers |
| 166 | 168 | * +-----+ |
| 167 | 169 | * | | x25/x26 |
| 168 | − * BPF fp register => -80:+-----+ | |
| 170 | + * BPF fp register => -80:+-----+ <= (BPF_FP) | |
| 169 | 171 | * | | |
| 170 | 172 | * | ... | BPF prog stack |
| 171 | 173 | * | | |
| 172 | − * | | | |
| 173 | − * current A64_SP => +-----+ | |
| 174 | + * +-----+ <= (BPF_FP - MAX_BPF_STACK) | |
| 175 | + * |RSVD | JIT scratchpad | |
| 176 | + * current A64_SP => +-----+ <= (BPF_FP - STACK_SIZE) | |
| 174 | 177 | * | | |
| 175 | 178 | * | ... | Function call stack |
| 176 | 179 | * | | |
| @@ -196,7 +199,7 @@ static void build_prologue(struct jit_ctx *ctx) · 19 unmodified lines | ||
| 196 | 199 | emit(A64_MOV(1, fp, A64_SP), ctx); |
| 197 | 200 | |
| 198 | 201 | /* Set up function call stack */ |
| 199 | − emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx); | |
| 202 | + emit(A64_SUB_I(1, A64_SP, A64_SP, STACK_SIZE), ctx); | |
| 200 | 203 | |
| 201 | 204 | /* Clear registers A and X */ |
| 202 | 205 | emit_a64_mov_i64(ra, 0, ctx); |
| @@ -213,13 +216,9 @@ static void build_epilogue(struct jit_ctx *ctx) · 10 unmodified lines | ||
| 213 | 216 | const u8 fp = bpf2a64[BPF_REG_FP]; |
| 214 | 217 | const u8 tmp1 = bpf2a64[TMP_REG_1]; |
| 215 | 218 | const u8 tmp2 = bpf2a64[TMP_REG_2]; |
| 216 | − int stack_size = MAX_BPF_STACK; | |
| 217 | − | |
| 218 | − stack_size += 4; /* extra for skb_copy_bits buffer */ | |
| 219 | − stack_size = STACK_ALIGN(stack_size); | |
| 220 | 219 | |
| 221 | 220 | /* We're done with BPF stack */ |
| 222 | − emit(A64_ADD_I(1, A64_SP, A64_SP, stack_size), ctx); | |
| 221 | + emit(A64_ADD_I(1, A64_SP, A64_SP, STACK_SIZE), ctx); | |
| 223 | 222 | |
| 224 | 223 | /* Restore fs (x25) and x26 */ |
| 225 | 224 | emit(A64_POP(fp, A64_R(26), A64_SP), ctx); |
| @@ -591,7 +590,25 @@ emit_cond_jmp: · 365 unmodified lines | ||
| 591 | 590 | case BPF_ST | BPF_MEM | BPF_H: |
| 592 | 591 | case BPF_ST | BPF_MEM | BPF_B: |
| 593 | 592 | case BPF_ST | BPF_MEM | BPF_DW: |
| 594 | − goto notyet; | |
| 593 | + /* Load imm to a register then store it */ | |
| 594 | + ctx->tmp_used = 1; | |
| 595 | + emit_a64_mov_i(1, tmp2, off, ctx); | |
| 596 | + emit_a64_mov_i(1, tmp, imm, ctx); | |
| 597 | + switch (BPF_SIZE(code)) { | |
| 598 | + case BPF_W: | |
| 599 | + emit(A64_STR32(tmp, dst, tmp2), ctx); | |
| 600 | + break; | |
| 601 | + case BPF_H: | |
| 602 | + emit(A64_STRH(tmp, dst, tmp2), ctx); | |
| 603 | + break; | |
| 604 | + case BPF_B: | |
| 605 | + emit(A64_STRB(tmp, dst, tmp2), ctx); | |
| 606 | + break; | |
| 607 | + case BPF_DW: | |
| 608 | + emit(A64_STR64(tmp, dst, tmp2), ctx); | |
| 609 | + break; | |
| 610 | + } | |
| 611 | + break; | |
| 595 | 612 | |
| 596 | 613 | /* STX: *(size *)(dst + off) = src */ |
| 597 | 614 | case BPF_STX | BPF_MEM | BPF_W: |
| @@ -658,7 +675,7 @@ emit_cond_jmp: · 60 unmodified lines | ||
| 658 | 675 | return -EINVAL; |
| 659 | 676 | } |
| 660 | 677 | emit_a64_mov_i64(r3, size, ctx); |
| 661 | − emit(A64_ADD_I(1, r4, fp, MAX_BPF_STACK), ctx); | |
| 678 | + emit(A64_SUB_I(1, r4, fp, STACK_SIZE), ctx); | |
| 662 | 679 | emit_a64_mov_i64(r5, (unsigned long)bpf_load_pointer, ctx); |
| 663 | 680 | emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); |
| 664 | 681 | emit(A64_MOV(1, A64_FP, A64_SP), ctx); |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 15 | 15 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 16 | 16 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 17 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 17 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 18 | 18 | * Copyright (C) 2009 Intel Corporation, <markus.t.metzger@intel.com> |
| 19 | 19 | * |
| 20 | 20 | * ppc: |
| @@ -98,7 +98,7 @@ static void __init mcf54xx_bootmem_alloc(void) · 97 unmodified lines | ||
| 98 | 98 | memstart = PAGE_ALIGN(_ramstart); |
| 99 | 99 | min_low_pfn = PFN_DOWN(_rambase); |
| 100 | 100 | start_pfn = PFN_DOWN(memstart); |
| 101 | − max_low_pfn = PFN_DOWN(_ramend); | |
| 101 | + max_pfn = max_low_pfn = PFN_DOWN(_ramend); | |
| 102 | 102 | high_memory = (void *)_ramend; |
| 103 | 103 | |
| 104 | 104 | m68k_virt_to_node_shift = fls(_ramend - _rambase - 1) - 6; |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #include <uapi/asm/unistd.h> |
| 5 | 5 | |
| 6 | 6 | |
| 7 | −#define NR_syscalls 375 | |
| 7 | +#define NR_syscalls 376 | |
| 8 | 8 | |
| 9 | 9 | #define __ARCH_WANT_OLD_READDIR |
| 10 | 10 | #define __ARCH_WANT_OLD_STAT |
| @@ -380,5 +380,6 @@ · 379 unmodified lines | ||
| 380 | 380 | #define __NR_sendmmsg 372 |
| 381 | 381 | #define __NR_userfaultfd 373 |
| 382 | 382 | #define __NR_membarrier 374 |
| 383 | +#define __NR_mlock2 375 | |
| 383 | 384 | |
| 384 | 385 | #endif /* _UAPI_ASM_M68K_UNISTD_H_ */ |
| @@ -238,11 +238,14 @@ void __init setup_arch(char **cmdline_p) · 237 unmodified lines | ||
| 238 | 238 | * Give all the memory to the bootmap allocator, tell it to put the |
| 239 | 239 | * boot mem_map at the start of memory. |
| 240 | 240 | */ |
| 241 | + min_low_pfn = PFN_DOWN(memory_start); | |
| 242 | + max_pfn = max_low_pfn = PFN_DOWN(memory_end); | |
| 243 | + | |
| 241 | 244 | bootmap_size = init_bootmem_node( |
| 242 | 245 | NODE_DATA(0), |
| 243 | − memory_start >> PAGE_SHIFT, /* map goes here */ | |
| 244 | − PAGE_OFFSET >> PAGE_SHIFT, /* 0 on coldfire */ | |
| 245 | − memory_end >> PAGE_SHIFT); | |
| 246 | + min_low_pfn, /* map goes here */ | |
| 247 | + PFN_DOWN(PAGE_OFFSET), | |
| 248 | + max_pfn); | |
| 246 | 249 | /* |
| 247 | 250 | * Free the usable memory, we have to make sure we do not free |
| 248 | 251 | * the bootmem bitmap so we then reserve it after freeing it :-) |
| @@ -395,3 +395,4 @@ ENTRY(sys_call_table) · 394 unmodified lines | ||
| 395 | 395 | .long sys_sendmmsg |
| 396 | 396 | .long sys_userfaultfd |
| 397 | 397 | .long sys_membarrier |
| 398 | + .long sys_mlock2 /* 375 */ | |
| @@ -250,7 +250,7 @@ void __init paging_init(void) · 249 unmodified lines | ||
| 250 | 250 | high_memory = phys_to_virt(max_addr); |
| 251 | 251 | |
| 252 | 252 | min_low_pfn = availmem >> PAGE_SHIFT; |
| 253 | − max_low_pfn = max_addr >> PAGE_SHIFT; | |
| 253 | + max_pfn = max_low_pfn = max_addr >> PAGE_SHIFT; | |
| 254 | 254 | |
| 255 | 255 | for (i = 0; i < m68k_num_memory; i++) { |
| 256 | 256 | addr = m68k_memory[i].addr; |
| @@ -118,13 +118,13 @@ static void __init sun3_bootmem_alloc(unsigned long memory_start, · 117 unmodified lines | ||
| 118 | 118 | memory_end = memory_end & PAGE_MASK; |
| 119 | 119 | |
| 120 | 120 | start_page = __pa(memory_start) >> PAGE_SHIFT; |
| 121 | − num_pages = __pa(memory_end) >> PAGE_SHIFT; | |
| 121 | + max_pfn = num_pages = __pa(memory_end) >> PAGE_SHIFT; | |
| 122 | 122 | |
| 123 | 123 | high_memory = (void *)memory_end; |
| 124 | 124 | availmem = memory_start; |
| 125 | 125 | |
| 126 | 126 | m68k_setup_node(0); |
| 127 | − availmem += init_bootmem_node(NODE_DATA(0), start_page, 0, num_pages); | |
| 127 | + availmem += init_bootmem(start_page, num_pages); | |
| 128 | 128 | availmem = (availmem + (PAGE_SIZE-1)) & PAGE_MASK; |
| 129 | 129 | |
| 130 | 130 | free_bootmem(__pa(availmem), memory_end - (availmem)); |
| @@ -145,7 +145,7 @@ static void *mips_dma_alloc_coherent(struct device *dev, size_t size, · 144 unmodified lines | ||
| 145 | 145 | |
| 146 | 146 | gfp = massage_gfp_flags(dev, gfp); |
| 147 | 147 | |
| 148 | − if (IS_ENABLED(CONFIG_DMA_CMA) && !(gfp & GFP_ATOMIC)) | |
| 148 | + if (IS_ENABLED(CONFIG_DMA_CMA) && gfpflags_allow_blocking(gfp)) | |
| 149 | 149 | page = dma_alloc_from_contiguous(dev, |
| 150 | 150 | count, get_order(size)); |
| 151 | 151 | if (!page) |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | * by the Free Software Foundation. |
| 12 | 12 | */ |
| 13 | 13 | |
| 14 | +#include <linux/delay.h> | |
| 14 | 15 | #include <linux/types.h> |
| 15 | 16 | #include <linux/pci.h> |
| 16 | 17 | #include <linux/io.h> |
| @@ -232,8 +233,7 @@ static int rt288x_pci_probe(struct platform_device *pdev) · 215 unmodified lines | ||
| 232 | 233 | ioport_resource.end = RT2880_PCI_IO_BASE + RT2880_PCI_IO_SIZE - 1; |
| 233 | 234 | |
| 234 | 235 | rt2880_pci_reg_write(0, RT2880_PCI_REG_PCICFG_ADDR); |
| 235 | − for (i = 0; i < 0xfffff; i++) | |
| 236 | − ; | |
| 236 | + udelay(1); | |
| 237 | 237 | |
| 238 | 238 | rt2880_pci_reg_write(0x79, RT2880_PCI_REG_ARBCTL); |
| 239 | 239 | rt2880_pci_reg_write(0x07FF0001, RT2880_PCI_REG_BAR0SETUP_ADDR); |
| @@ -10,6 +10,8 @@ · 9 unmodified lines | ||
| 10 | 10 | * option) any later version. |
| 11 | 11 | */ |
| 12 | 12 | |
| 13 | +#include <linux/delay.h> | |
| 14 | + | |
| 13 | 15 | #include <asm/bootinfo.h> |
| 14 | 16 | #include <asm/cacheflush.h> |
| 15 | 17 | #include <asm/idle.h> |
| @@ -77,7 +79,7 @@ void msp7120_reset(void) · 61 unmodified lines | ||
| 77 | 79 | */ |
| 78 | 80 | |
| 79 | 81 | /* Wait a bit for the DDRC to settle */ |
| 80 | − for (i = 0; i < 100000000; i++); | |
| 82 | + mdelay(125); | |
| 81 | 83 | |
| 82 | 84 | #if defined(CONFIG_PMC_MSP7120_GW) |
| 83 | 85 | /* |
| @@ -3,6 +3,8 @@ · 2 unmodified lines | ||
| 3 | 3 | * |
| 4 | 4 | * Reset a SNI machine. |
| 5 | 5 | */ |
| 6 | +#include <linux/delay.h> | |
| 7 | + | |
| 6 | 8 | #include <asm/io.h> |
| 7 | 9 | #include <asm/reboot.h> |
| 8 | 10 | #include <asm/sni.h> |
| @@ -32,9 +34,9 @@ void sni_machine_restart(char *command) · 23 unmodified lines | ||
| 32 | 34 | for (;;) { |
| 33 | 35 | for (i = 0; i < 100; i++) { |
| 34 | 36 | kb_wait(); |
| 35 | − for (j = 0; j < 100000 ; j++) | |
| 36 | − /* nothing */; | |
| 37 | + udelay(50); | |
| 37 | 38 | outb_p(0xfe, 0x64); /* pulse reset low */ |
| 39 | + udelay(50); | |
| 38 | 40 | } |
| 39 | 41 | } |
| 40 | 42 | } |
| @@ -1,6 +1,7 @@ | ||
| 1 | 1 | config MN10300 |
| 2 | 2 | def_bool y |
| 3 | 3 | select HAVE_OPROFILE |
| 4 | + select HAVE_UID16 | |
| 4 | 5 | select GENERIC_IRQ_SHOW |
| 5 | 6 | select ARCH_WANT_IPC_PARSE_VERSION |
| 6 | 7 | select HAVE_ARCH_TRACEHOOK |
| @@ -37,9 +38,6 @@ config HIGHMEM · 30 unmodified lines | ||
| 37 | 38 | config NUMA |
| 38 | 39 | def_bool n |
| 39 | 40 | |
| 40 | −config UID16 | |
| 41 | − def_bool y | |
| 42 | − | |
| 43 | 41 | config RWSEM_GENERIC_SPINLOCK |
| 44 | 42 | def_bool y |
| 45 | 43 | |
| @@ -23,22 +23,6 @@ static void __flush_dcache(unsigned long start, unsigned long end) · 22 unmodified lines | ||
| 23 | 23 | end += (cpuinfo.dcache_line_size - 1); |
| 24 | 24 | end &= ~(cpuinfo.dcache_line_size - 1); |
| 25 | 25 | |
| 26 | − for (addr = start; addr < end; addr += cpuinfo.dcache_line_size) { | |
| 27 | − __asm__ __volatile__ (" flushda 0(%0)\n" | |
| 28 | − : /* Outputs */ | |
| 29 | − : /* Inputs */ "r"(addr) | |
| 30 | − /* : No clobber */); | |
| 31 | − } | |
| 32 | −} | |
| 33 | − | |
| 34 | −static void __flush_dcache_all(unsigned long start, unsigned long end) | |
| 35 | −{ | |
| 36 | − unsigned long addr; | |
| 37 | − | |
| 38 | − start &= ~(cpuinfo.dcache_line_size - 1); | |
| 39 | − end += (cpuinfo.dcache_line_size - 1); | |
| 40 | − end &= ~(cpuinfo.dcache_line_size - 1); | |
| 41 | − | |
| 42 | 26 | if (end > start + cpuinfo.dcache_size) |
| 43 | 27 | end = start + cpuinfo.dcache_size; |
| 44 | 28 | |
| @@ -112,7 +96,7 @@ static void flush_aliases(struct address_space *mapping, struct page *page) · 67 unmodified lines | ||
| 112 | 96 | |
| 113 | 97 | void flush_cache_all(void) |
| 114 | 98 | { |
| 115 | − __flush_dcache_all(0, cpuinfo.dcache_size); | |
| 99 | + __flush_dcache(0, cpuinfo.dcache_size); | |
| 116 | 100 | __flush_icache(0, cpuinfo.icache_size); |
| 117 | 101 | } |
| 118 | 102 | |
| @@ -182,7 +166,7 @@ void __flush_dcache_page(struct address_space *mapping, struct page *page) · 63 unmodified lines | ||
| 182 | 166 | */ |
| 183 | 167 | unsigned long start = (unsigned long)page_address(page); |
| 184 | 168 | |
| 185 | − __flush_dcache_all(start, start + PAGE_SIZE); | |
| 169 | + __flush_dcache(start, start + PAGE_SIZE); | |
| 186 | 170 | } |
| 187 | 171 | |
| 188 | 172 | void flush_dcache_page(struct page *page) |
| @@ -268,7 +252,7 @@ void copy_from_user_page(struct vm_area_struct *vma, struct page *page, · 79 unmodified lines | ||
| 268 | 252 | { |
| 269 | 253 | flush_cache_page(vma, user_vaddr, page_to_pfn(page)); |
| 270 | 254 | memcpy(dst, src, len); |
| 271 | − __flush_dcache_all((unsigned long)src, (unsigned long)src + len); | |
| 255 | + __flush_dcache((unsigned long)src, (unsigned long)src + len); | |
| 272 | 256 | if (vma->vm_flags & VM_EXEC) |
| 273 | 257 | __flush_icache((unsigned long)src, (unsigned long)src + len); |
| 274 | 258 | } |
| @@ -279,7 +263,7 @@ void copy_to_user_page(struct vm_area_struct *vma, struct page *page, · 4 unmodified lines | ||
| 279 | 263 | { |
| 280 | 264 | flush_cache_page(vma, user_vaddr, page_to_pfn(page)); |
| 281 | 265 | memcpy(dst, src, len); |
| 282 | − __flush_dcache_all((unsigned long)dst, (unsigned long)dst + len); | |
| 266 | + __flush_dcache((unsigned long)dst, (unsigned long)dst + len); | |
| 283 | 267 | if (vma->vm_flags & VM_EXEC) |
| 284 | 268 | __flush_icache((unsigned long)dst, (unsigned long)dst + len); |
| 285 | 269 | } |
| @@ -372,7 +372,8 @@ static inline pte_t pte_mkspecial(pte_t pte) { return pte; } · 371 unmodified lines | ||
| 372 | 372 | */ |
| 373 | 373 | #ifdef CONFIG_HUGETLB_PAGE |
| 374 | 374 | #define pte_huge(pte) (pte_val(pte) & _PAGE_HUGE) |
| 375 | −#define pte_mkhuge(pte) (__pte(pte_val(pte) | _PAGE_HUGE)) | |
| 375 | +#define pte_mkhuge(pte) (__pte(pte_val(pte) | \ | |
| 376 | + (parisc_requires_coherency() ? 0 : _PAGE_HUGE))) | |
| 376 | 377 | #else |
| 377 | 378 | #define pte_huge(pte) (0) |
| 378 | 379 | #define pte_mkhuge(pte) (pte) |
| @@ -360,8 +360,9 @@ · 359 unmodified lines | ||
| 360 | 360 | #define __NR_execveat (__NR_Linux + 342) |
| 361 | 361 | #define __NR_membarrier (__NR_Linux + 343) |
| 362 | 362 | #define __NR_userfaultfd (__NR_Linux + 344) |
| 363 | +#define __NR_mlock2 (__NR_Linux + 345) | |
| 363 | 364 | |
| 364 | −#define __NR_Linux_syscalls (__NR_userfaultfd + 1) | |
| 365 | +#define __NR_Linux_syscalls (__NR_mlock2 + 1) | |
| 365 | 366 | |
| 366 | 367 | |
| 367 | 368 | #define __IGNORE_select /* newselect */ |
| @@ -171,24 +171,6 @@ void pcibios_set_master(struct pci_dev *dev) · 170 unmodified lines | ||
| 171 | 171 | } |
| 172 | 172 | |
| 173 | 173 | |
| 174 | −void __init pcibios_init_bus(struct pci_bus *bus) | |
| 175 | −{ | |
| 176 | − struct pci_dev *dev = bus->self; | |
| 177 | − unsigned short bridge_ctl; | |
| 178 | − | |
| 179 | − /* We deal only with pci controllers and pci-pci bridges. */ | |
| 180 | − if (!dev || (dev->class >> 8) != PCI_CLASS_BRIDGE_PCI) | |
| 181 | − return; | |
| 182 | − | |
| 183 | − /* PCI-PCI bridge - set the cache line and default latency | |
| 184 | − (32) for primary and secondary buses. */ | |
| 185 | − pci_write_config_byte(dev, PCI_SEC_LATENCY_TIMER, 32); | |
| 186 | − | |
| 187 | − pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &bridge_ctl); | |
| 188 | − bridge_ctl |= PCI_BRIDGE_CTL_PARITY | PCI_BRIDGE_CTL_SERR; | |
| 189 | − pci_write_config_word(dev, PCI_BRIDGE_CONTROL, bridge_ctl); | |
| 190 | −} | |
| 191 | − | |
| 192 | 174 | /* |
| 193 | 175 | * pcibios align resources() is called every time generic PCI code |
| 194 | 176 | * wants to generate a new address. The process of looking for |
| @@ -440,6 +440,7 @@ · 439 unmodified lines | ||
| 440 | 440 | ENTRY_COMP(execveat) |
| 441 | 441 | ENTRY_SAME(membarrier) |
| 442 | 442 | ENTRY_SAME(userfaultfd) |
| 443 | + ENTRY_SAME(mlock2) /* 345 */ | |
| 443 | 444 | |
| 444 | 445 | |
| 445 | 446 | .ifne (. - 90b) - (__NR_Linux_syscalls * (91b - 90b)) |
| @@ -227,23 +227,15 @@ · 226 unmodified lines | ||
| 227 | 227 | reg = <0x520 0x20>; |
| 228 | 228 | |
| 229 | 229 | phy0: ethernet-phy@1f { |
| 230 | − interrupt-parent = <&mpic>; | |
| 231 | − interrupts = <10 1>; | |
| 232 | 230 | reg = <0x1f>; |
| 233 | 231 | }; |
| 234 | 232 | phy1: ethernet-phy@0 { |
| 235 | − interrupt-parent = <&mpic>; | |
| 236 | − interrupts = <10 1>; | |
| 237 | 233 | reg = <0>; |
| 238 | 234 | }; |
| 239 | 235 | phy2: ethernet-phy@1 { |
| 240 | − interrupt-parent = <&mpic>; | |
| 241 | − interrupts = <10 1>; | |
| 242 | 236 | reg = <1>; |
| 243 | 237 | }; |
| 244 | 238 | phy3: ethernet-phy@2 { |
| 245 | − interrupt-parent = <&mpic>; | |
| 246 | − interrupts = <10 1>; | |
| 247 | 239 | reg = <2>; |
| 248 | 240 | }; |
| 249 | 241 | tbi0: tbi-phy@11 { |
| @@ -108,6 +108,7 @@ · 107 unmodified lines | ||
| 108 | 108 | #define MSR_TS_T __MASK(MSR_TS_T_LG) /* Transaction Transactional */ |
| 109 | 109 | #define MSR_TS_MASK (MSR_TS_T | MSR_TS_S) /* Transaction State bits */ |
| 110 | 110 | #define MSR_TM_ACTIVE(x) (((x) & MSR_TS_MASK) != 0) /* Transaction active? */ |
| 111 | +#define MSR_TM_RESV(x) (((x) & MSR_TS_MASK) == MSR_TS_MASK) /* Reserved */ | |
| 111 | 112 | #define MSR_TM_TRANSACTIONAL(x) (((x) & MSR_TS_MASK) == MSR_TS_T) |
| 112 | 113 | #define MSR_TM_SUSPENDED(x) (((x) & MSR_TS_MASK) == MSR_TS_S) |
| 113 | 114 | |
| @@ -590,16 +590,10 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus) · 589 unmodified lines | ||
| 590 | 590 | eeh_ops->configure_bridge(pe); |
| 591 | 591 | eeh_pe_restore_bars(pe); |
| 592 | 592 | |
| 593 | − /* | |
| 594 | − * If it's PHB PE, the frozen state on all available PEs should have | |
| 595 | − * been cleared by the PHB reset. Otherwise, we unfreeze the PE and its | |
| 596 | − * child PEs because they might be in frozen state. | |
| 597 | − */ | |
| 598 | − if (!(pe->type & EEH_PE_PHB)) { | |
| 599 | − rc = eeh_clear_pe_frozen_state(pe, false); | |
| 600 | − if (rc) | |
| 601 | − return rc; | |
| 602 | − } | |
| 593 | + /* Clear frozen state */ | |
| 594 | + rc = eeh_clear_pe_frozen_state(pe, false); | |
| 595 | + if (rc) | |
| 596 | + return rc; | |
| 603 | 597 | |
| 604 | 598 | /* Give the system 5 seconds to finish running the user-space |
| 605 | 599 | * hotplug shutdown scripts, e.g. ifdown for ethernet. Yes, |
| @@ -551,6 +551,24 @@ static void tm_reclaim_thread(struct thread_struct *thr, · 550 unmodified lines | ||
| 551 | 551 | msr_diff &= MSR_FP | MSR_VEC | MSR_VSX | MSR_FE0 | MSR_FE1; |
| 552 | 552 | } |
| 553 | 553 | |
| 554 | + /* | |
| 555 | + * Use the current MSR TM suspended bit to track if we have | |
| 556 | + * checkpointed state outstanding. | |
| 557 | + * On signal delivery, we'd normally reclaim the checkpointed | |
| 558 | + * state to obtain stack pointer (see:get_tm_stackpointer()). | |
| 559 | + * This will then directly return to userspace without going | |
| 560 | + * through __switch_to(). However, if the stack frame is bad, | |
| 561 | + * we need to exit this thread which calls __switch_to() which | |
| 562 | + * will again attempt to reclaim the already saved tm state. | |
| 563 | + * Hence we need to check that we've not already reclaimed | |
| 564 | + * this state. | |
| 565 | + * We do this using the current MSR, rather tracking it in | |
| 566 | + * some specific thread_struct bit, as it has the additional | |
| 567 | + * benifit of checking for a potential TM bad thing exception. | |
| 568 | + */ | |
| 569 | + if (!MSR_TM_SUSPENDED(mfmsr())) | |
| 570 | + return; | |
| 571 | + | |
| 554 | 572 | tm_reclaim(thr, thr->regs->msr, cause); |
| 555 | 573 | |
| 556 | 574 | /* Having done the reclaim, we now have the checkpointed |
| @@ -875,6 +875,15 @@ static long restore_tm_user_regs(struct pt_regs *regs, · 874 unmodified lines | ||
| 875 | 875 | return 1; |
| 876 | 876 | #endif /* CONFIG_SPE */ |
| 877 | 877 | |
| 878 | + /* Get the top half of the MSR from the user context */ | |
| 879 | + if (__get_user(msr_hi, &tm_sr->mc_gregs[PT_MSR])) | |
| 880 | + return 1; | |
| 881 | + msr_hi <<= 32; | |
| 882 | + /* If TM bits are set to the reserved value, it's an invalid context */ | |
| 883 | + if (MSR_TM_RESV(msr_hi)) | |
| 884 | + return 1; | |
| 885 | + /* Pull in the MSR TM bits from the user context */ | |
| 886 | + regs->msr = (regs->msr & ~MSR_TS_MASK) | (msr_hi & MSR_TS_MASK); | |
| 878 | 887 | /* Now, recheckpoint. This loads up all of the checkpointed (older) |
| 879 | 888 | * registers, including FP and V[S]Rs. After recheckpointing, the |
| 880 | 889 | * transactional versions should be loaded. |
| @@ -884,11 +893,6 @@ static long restore_tm_user_regs(struct pt_regs *regs, · 3 unmodified lines | ||
| 884 | 893 | current->thread.tm_texasr |= TEXASR_FS; |
| 885 | 894 | /* This loads the checkpointed FP/VEC state, if used */ |
| 886 | 895 | tm_recheckpoint(¤t->thread, msr); |
| 887 | − /* Get the top half of the MSR */ | |
| 888 | − if (__get_user(msr_hi, &tm_sr->mc_gregs[PT_MSR])) | |
| 889 | − return 1; | |
| 890 | − /* Pull in MSR TM from user context */ | |
| 891 | − regs->msr = (regs->msr & ~MSR_TS_MASK) | ((msr_hi<<32) & MSR_TS_MASK); | |
| 892 | 896 | |
| 893 | 897 | /* This loads the speculative FP/VEC state, if used */ |
| 894 | 898 | if (msr & MSR_FP) { |
| @@ -438,6 +438,10 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, · 437 unmodified lines | ||
| 438 | 438 | |
| 439 | 439 | /* get MSR separately, transfer the LE bit if doing signal return */ |
| 440 | 440 | err |= __get_user(msr, &sc->gp_regs[PT_MSR]); |
| 441 | + /* Don't allow reserved mode. */ | |
| 442 | + if (MSR_TM_RESV(msr)) | |
| 443 | + return -EINVAL; | |
| 444 | + | |
| 441 | 445 | /* pull in MSR TM from user context */ |
| 442 | 446 | regs->msr = (regs->msr & ~MSR_TS_MASK) | (msr & MSR_TS_MASK); |
| 443 | 447 | |
| @@ -43,11 +43,34 @@ static unsigned int opal_irq_count; · 42 unmodified lines | ||
| 43 | 43 | static unsigned int *opal_irqs; |
| 44 | 44 | |
| 45 | 45 | static void opal_handle_irq_work(struct irq_work *work); |
| 46 | −static __be64 last_outstanding_events; | |
| 46 | +static u64 last_outstanding_events; | |
| 47 | 47 | static struct irq_work opal_event_irq_work = { |
| 48 | 48 | .func = opal_handle_irq_work, |
| 49 | 49 | }; |
| 50 | 50 | |
| 51 | +void opal_handle_events(uint64_t events) | |
| 52 | +{ | |
| 53 | + int virq, hwirq = 0; | |
| 54 | + u64 mask = opal_event_irqchip.mask; | |
| 55 | + | |
| 56 | + if (!in_irq() && (events & mask)) { | |
| 57 | + last_outstanding_events = events; | |
| 58 | + irq_work_queue(&opal_event_irq_work); | |
| 59 | + return; | |
| 60 | + } | |
| 61 | + | |
| 62 | + while (events & mask) { | |
| 63 | + hwirq = fls64(events) - 1; | |
| 64 | + if (BIT_ULL(hwirq) & mask) { | |
| 65 | + virq = irq_find_mapping(opal_event_irqchip.domain, | |
| 66 | + hwirq); | |
| 67 | + if (virq) | |
| 68 | + generic_handle_irq(virq); | |
| 69 | + } | |
| 70 | + events &= ~BIT_ULL(hwirq); | |
| 71 | + } | |
| 72 | +} | |
| 73 | + | |
| 51 | 74 | static void opal_event_mask(struct irq_data *d) |
| 52 | 75 | { |
| 53 | 76 | clear_bit(d->hwirq, &opal_event_irqchip.mask); |
| @@ -55,12 +78,12 @@ static void opal_event_mask(struct irq_data *d) · 1 unmodified line | ||
| 55 | 78 | |
| 56 | 79 | static void opal_event_unmask(struct irq_data *d) |
| 57 | 80 | { |
| 81 | + __be64 events; | |
| 82 | + | |
| 58 | 83 | set_bit(d->hwirq, &opal_event_irqchip.mask); |
| 59 | 84 | |
| 60 | − opal_poll_events(&last_outstanding_events); | |
| 61 | − if (last_outstanding_events & opal_event_irqchip.mask) | |
| 62 | − /* Need to retrigger the interrupt */ | |
| 63 | − irq_work_queue(&opal_event_irq_work); | |
| 85 | + opal_poll_events(&events); | |
| 86 | + opal_handle_events(be64_to_cpu(events)); | |
| 64 | 87 | } |
| 65 | 88 | |
| 66 | 89 | static int opal_event_set_type(struct irq_data *d, unsigned int flow_type) |
| @@ -96,29 +119,6 @@ static int opal_event_map(struct irq_domain *d, unsigned int irq, · 29 unmodified lines | ||
| 96 | 119 | return 0; |
| 97 | 120 | } |
| 98 | 121 | |
| 99 | −void opal_handle_events(uint64_t events) | |
| 100 | −{ | |
| 101 | − int virq, hwirq = 0; | |
| 102 | − u64 mask = opal_event_irqchip.mask; | |
| 103 | − | |
| 104 | − if (!in_irq() && (events & mask)) { | |
| 105 | − last_outstanding_events = events; | |
| 106 | − irq_work_queue(&opal_event_irq_work); | |
| 107 | − return; | |
| 108 | − } | |
| 109 | − | |
| 110 | − while (events & mask) { | |
| 111 | − hwirq = fls64(events) - 1; | |
| 112 | − if (BIT_ULL(hwirq) & mask) { | |
| 113 | − virq = irq_find_mapping(opal_event_irqchip.domain, | |
| 114 | − hwirq); | |
| 115 | − if (virq) | |
| 116 | − generic_handle_irq(virq); | |
| 117 | − } | |
| 118 | − events &= ~BIT_ULL(hwirq); | |
| 119 | − } | |
| 120 | −} | |
| 121 | − | |
| 122 | 122 | static irqreturn_t opal_interrupt(int irq, void *data) |
| 123 | 123 | { |
| 124 | 124 | __be64 events; |
| @@ -131,7 +131,7 @@ static irqreturn_t opal_interrupt(int irq, void *data) · 6 unmodified lines | ||
| 131 | 131 | |
| 132 | 132 | static void opal_handle_irq_work(struct irq_work *work) |
| 133 | 133 | { |
| 134 | − opal_handle_events(be64_to_cpu(last_outstanding_events)); | |
| 134 | + opal_handle_events(last_outstanding_events); | |
| 135 | 135 | } |
| 136 | 136 | |
| 137 | 137 | static int opal_event_match(struct irq_domain *h, struct device_node *node, |
| @@ -278,7 +278,7 @@ · 277 unmodified lines | ||
| 278 | 278 | #define __NR_fsetxattr 256 |
| 279 | 279 | #define __NR_getxattr 257 |
| 280 | 280 | #define __NR_lgetxattr 258 |
| 281 | −#define __NR_fgetxattr 269 | |
| 281 | +#define __NR_fgetxattr 259 | |
| 282 | 282 | #define __NR_listxattr 260 |
| 283 | 283 | #define __NR_llistxattr 261 |
| 284 | 284 | #define __NR_flistxattr 262 |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 11 | 11 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 12 | 12 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 13 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 13 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 14 | 14 | * Copyright (C) 2009 Intel Corporation, <markus.t.metzger@intel.com> |
| 15 | 15 | * |
| 16 | 16 | * ppc: |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 10 | 10 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 11 | 11 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 12 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 12 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 13 | 13 | */ |
| 14 | 14 | |
| 15 | 15 | #include <linux/perf_event.h> |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 22 | 22 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 23 | 23 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 24 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 24 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 25 | 25 | * Copyright (C) 2009 Intel Corporation, <markus.t.metzger@intel.com> |
| 26 | 26 | * Copyright (C) 2009 Google, Inc., Stephane Eranian |
| 27 | 27 | */ |
| @@ -131,7 +131,7 @@ export LDS_ELF_FORMAT := $(ELF_FORMAT) · 130 unmodified lines | ||
| 131 | 131 | # The wrappers will select whether using "malloc" or the kernel allocator. |
| 132 | 132 | LINK_WRAPS = -Wl,--wrap,malloc -Wl,--wrap,free -Wl,--wrap,calloc |
| 133 | 133 | |
| 134 | −LD_FLAGS_CMDLINE = $(foreach opt,$(LDFLAGS),-Wl,$(opt)) -lrt | |
| 134 | +LD_FLAGS_CMDLINE = $(foreach opt,$(LDFLAGS),-Wl,$(opt)) | |
| 135 | 135 | |
| 136 | 136 | # Used by link-vmlinux.sh which has special support for um link |
| 137 | 137 | export CFLAGS_vmlinux := $(LINK-y) $(LINK_WRAPS) $(LD_FLAGS_CMDLINE) |
| @@ -249,21 +249,23 @@ void close_addr(unsigned char *addr, unsigned char *netmask, void *arg) · 248 unmodified lines | ||
| 249 | 249 | |
| 250 | 250 | char *split_if_spec(char *str, ...) |
| 251 | 251 | { |
| 252 | − char **arg, *end; | |
| 252 | + char **arg, *end, *ret = NULL; | |
| 253 | 253 | va_list ap; |
| 254 | 254 | |
| 255 | 255 | va_start(ap, str); |
| 256 | 256 | while ((arg = va_arg(ap, char **)) != NULL) { |
| 257 | 257 | if (*str == '\0') |
| 258 | − return NULL; | |
| 258 | + goto out; | |
| 259 | 259 | end = strchr(str, ','); |
| 260 | 260 | if (end != str) |
| 261 | 261 | *arg = str; |
| 262 | 262 | if (end == NULL) |
| 263 | − return NULL; | |
| 263 | + goto out; | |
| 264 | 264 | *end++ = '\0'; |
| 265 | 265 | str = end; |
| 266 | 266 | } |
| 267 | + ret = str; | |
| 268 | +out: | |
| 267 | 269 | va_end(ap); |
| 268 | − return str; | |
| 270 | + return ret; | |
| 269 | 271 | } |
| @@ -69,7 +69,7 @@ void do_signal(struct pt_regs *regs) · 68 unmodified lines | ||
| 69 | 69 | struct ksignal ksig; |
| 70 | 70 | int handled_sig = 0; |
| 71 | 71 | |
| 72 | − while (get_signal(&ksig)) { | |
| 72 | + if (get_signal(&ksig)) { | |
| 73 | 73 | handled_sig = 1; |
| 74 | 74 | /* Whee! Actually deliver the signal. */ |
| 75 | 75 | handle_signal(&ksig, regs); |
| @@ -23,7 +23,6 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <stdarg.h> |
| 24 | 24 | #include <linux/types.h> |
| 25 | 25 | #include <linux/edd.h> |
| 26 | −#include <asm/boot.h> | |
| 27 | 26 | #include <asm/setup.h> |
| 28 | 27 | #include "bitops.h" |
| 29 | 28 | #include "ctype.h" |
| @@ -19,6 +19,8 @@ · 18 unmodified lines | ||
| 19 | 19 | #include "video.h" |
| 20 | 20 | #include "vesa.h" |
| 21 | 21 | |
| 22 | +#include <uapi/asm/boot.h> | |
| 23 | + | |
| 22 | 24 | /* |
| 23 | 25 | * Common variables |
| 24 | 26 | */ |
| @@ -13,6 +13,8 @@ · 12 unmodified lines | ||
| 13 | 13 | * Select video mode |
| 14 | 14 | */ |
| 15 | 15 | |
| 16 | +#include <uapi/asm/boot.h> | |
| 17 | + | |
| 16 | 18 | #include "boot.h" |
| 17 | 19 | #include "video.h" |
| 18 | 20 | #include "vesa.h" |
| @@ -509,6 +509,17 @@ END(irq_entries_start) · 508 unmodified lines | ||
| 509 | 509 | * tracking that we're in kernel mode. |
| 510 | 510 | */ |
| 511 | 511 | SWAPGS |
| 512 | + | |
| 513 | + /* | |
| 514 | + * We need to tell lockdep that IRQs are off. We can't do this until | |
| 515 | + * we fix gsbase, and we should do it before enter_from_user_mode | |
| 516 | + * (which can take locks). Since TRACE_IRQS_OFF idempotent, | |
| 517 | + * the simplest way to handle it is to just call it twice if | |
| 518 | + * we enter from user mode. There's no reason to optimize this since | |
| 519 | + * TRACE_IRQS_OFF is a no-op if lockdep is off. | |
| 520 | + */ | |
| 521 | + TRACE_IRQS_OFF | |
| 522 | + | |
| 512 | 523 | #ifdef CONFIG_CONTEXT_TRACKING |
| 513 | 524 | call enter_from_user_mode |
| 514 | 525 | #endif |
| @@ -1049,12 +1060,18 @@ ENTRY(error_entry) · 534 unmodified lines | ||
| 1049 | 1060 | SWAPGS |
| 1050 | 1061 | |
| 1051 | 1062 | .Lerror_entry_from_usermode_after_swapgs: |
| 1063 | + /* | |
| 1064 | + * We need to tell lockdep that IRQs are off. We can't do this until | |
| 1065 | + * we fix gsbase, and we should do it before enter_from_user_mode | |
| 1066 | + * (which can take locks). | |
| 1067 | + */ | |
| 1068 | + TRACE_IRQS_OFF | |
| 1052 | 1069 | #ifdef CONFIG_CONTEXT_TRACKING |
| 1053 | 1070 | call enter_from_user_mode |
| 1054 | 1071 | #endif |
| 1072 | + ret | |
| 1055 | 1073 | |
| 1056 | 1074 | .Lerror_entry_done: |
| 1057 | − | |
| 1058 | 1075 | TRACE_IRQS_OFF |
| 1059 | 1076 | ret |
| 1060 | 1077 | |
| @@ -9,19 +9,21 @@ · 8 unmodified lines | ||
| 9 | 9 | #define PAGE_SIZE (_AC(1,UL) << PAGE_SHIFT) |
| 10 | 10 | #define PAGE_MASK (~(PAGE_SIZE-1)) |
| 11 | 11 | |
| 12 | +#define PMD_PAGE_SIZE (_AC(1, UL) << PMD_SHIFT) | |
| 13 | +#define PMD_PAGE_MASK (~(PMD_PAGE_SIZE-1)) | |
| 14 | + | |
| 15 | +#define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT) | |
| 16 | +#define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1)) | |
| 17 | + | |
| 12 | 18 | #define __PHYSICAL_MASK ((phys_addr_t)((1ULL << __PHYSICAL_MASK_SHIFT) - 1)) |
| 13 | 19 | #define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1) |
| 14 | 20 | |
| 15 | −/* Cast PAGE_MASK to a signed type so that it is sign-extended if | |
| 21 | +/* Cast *PAGE_MASK to a signed type so that it is sign-extended if | |
| 16 | 22 | virtual addresses are 32-bits but physical addresses are larger |
| 17 | 23 | (ie, 32-bit PAE). */ |
| 18 | 24 | #define PHYSICAL_PAGE_MASK (((signed long)PAGE_MASK) & __PHYSICAL_MASK) |
| 19 | − | |
| 20 | −#define PMD_PAGE_SIZE (_AC(1, UL) << PMD_SHIFT) | |
| 21 | −#define PMD_PAGE_MASK (~(PMD_PAGE_SIZE-1)) | |
| 22 | − | |
| 23 | −#define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT) | |
| 24 | −#define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1)) | |
| 25 | +#define PHYSICAL_PMD_PAGE_MASK (((signed long)PMD_PAGE_MASK) & __PHYSICAL_MASK) | |
| 26 | +#define PHYSICAL_PUD_PAGE_MASK (((signed long)PUD_PAGE_MASK) & __PHYSICAL_MASK) | |
| 25 | 27 | |
| 26 | 28 | #define HPAGE_SHIFT PMD_SHIFT |
| 27 | 29 | #define HPAGE_SIZE (_AC(1,UL) << HPAGE_SHIFT) |
| @@ -279,17 +279,14 @@ static inline pmdval_t native_pmd_val(pmd_t pmd) · 278 unmodified lines | ||
| 279 | 279 | static inline pudval_t pud_pfn_mask(pud_t pud) |
| 280 | 280 | { |
| 281 | 281 | if (native_pud_val(pud) & _PAGE_PSE) |
| 282 | − return PUD_PAGE_MASK & PHYSICAL_PAGE_MASK; | |
| 282 | + return PHYSICAL_PUD_PAGE_MASK; | |
| 283 | 283 | else |
| 284 | 284 | return PTE_PFN_MASK; |
| 285 | 285 | } |
| 286 | 286 | |
| 287 | 287 | static inline pudval_t pud_flags_mask(pud_t pud) |
| 288 | 288 | { |
| 289 | − if (native_pud_val(pud) & _PAGE_PSE) | |
| 290 | − return ~(PUD_PAGE_MASK & (pudval_t)PHYSICAL_PAGE_MASK); | |
| 291 | − else | |
| 292 | − return ~PTE_PFN_MASK; | |
| 289 | + return ~pud_pfn_mask(pud); | |
| 293 | 290 | } |
| 294 | 291 | |
| 295 | 292 | static inline pudval_t pud_flags(pud_t pud) |
| @@ -300,17 +297,14 @@ static inline pudval_t pud_flags(pud_t pud) · 4 unmodified lines | ||
| 300 | 297 | static inline pmdval_t pmd_pfn_mask(pmd_t pmd) |
| 301 | 298 | { |
| 302 | 299 | if (native_pmd_val(pmd) & _PAGE_PSE) |
| 303 | − return PMD_PAGE_MASK & PHYSICAL_PAGE_MASK; | |
| 300 | + return PHYSICAL_PMD_PAGE_MASK; | |
| 304 | 301 | else |
| 305 | 302 | return PTE_PFN_MASK; |
| 306 | 303 | } |
| 307 | 304 | |
| 308 | 305 | static inline pmdval_t pmd_flags_mask(pmd_t pmd) |
| 309 | 306 | { |
| 310 | − if (native_pmd_val(pmd) & _PAGE_PSE) | |
| 311 | − return ~(PMD_PAGE_MASK & (pmdval_t)PHYSICAL_PAGE_MASK); | |
| 312 | − else | |
| 313 | − return ~PTE_PFN_MASK; | |
| 307 | + return ~pmd_pfn_mask(pmd); | |
| 314 | 308 | } |
| 315 | 309 | |
| 316 | 310 | static inline pmdval_t pmd_flags(pmd_t pmd) |
| @@ -1,7 +1,6 @@ | ||
| 1 | 1 | #ifndef _ASM_X86_PLATFORM_H |
| 2 | 2 | #define _ASM_X86_PLATFORM_H |
| 3 | 3 | |
| 4 | −#include <asm/pgtable_types.h> | |
| 5 | 4 | #include <asm/bootparam.h> |
| 6 | 5 | |
| 7 | 6 | struct mpc_bus; |
| @@ -698,3 +698,4 @@ int __init microcode_init(void) · 697 unmodified lines | ||
| 698 | 698 | return error; |
| 699 | 699 | |
| 700 | 700 | } |
| 701 | +late_initcall(microcode_init); | |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 6 | 6 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 7 | 7 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 8 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 8 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 9 | 9 | * Copyright (C) 2009 Intel Corporation, <markus.t.metzger@intel.com> |
| 10 | 10 | * Copyright (C) 2009 Google, Inc., Stephane Eranian |
| 11 | 11 | * |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * Copyright (C) 2008-2009 Red Hat, Inc., Ingo Molnar |
| 6 | 6 | * Copyright (C) 2009 Jaswinder Singh Rajput |
| 7 | 7 | * Copyright (C) 2009 Advanced Micro Devices, Inc., Robert Richter |
| 8 | − * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 8 | + * Copyright (C) 2008-2009 Red Hat, Inc., Peter Zijlstra | |
| 9 | 9 | * Copyright (C) 2009 Intel Corporation, <markus.t.metzger@intel.com> |
| 10 | 10 | * Copyright (C) 2009 Google, Inc., Stephane Eranian |
| 11 | 11 | * |
| @@ -387,7 +387,7 @@ struct cpu_hw_events { · 375 unmodified lines | ||
| 387 | 387 | /* Check flags and event code/umask, and set the HSW N/A flag */ |
| 388 | 388 | #define INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_NA(code, n) \ |
| 389 | 389 | __EVENT_CONSTRAINT(code, n, \ |
| 390 | − INTEL_ARCH_EVENT_MASK|INTEL_ARCH_EVENT_MASK, \ | |
| 390 | + INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \ | |
| 391 | 391 | HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_NA_HSW) |
| 392 | 392 | |
| 393 | 393 | |
| @@ -627,6 +627,7 @@ struct x86_perf_task_context { · 233 unmodified lines | ||
| 627 | 627 | u64 lbr_from[MAX_LBR_ENTRIES]; |
| 628 | 628 | u64 lbr_to[MAX_LBR_ENTRIES]; |
| 629 | 629 | u64 lbr_info[MAX_LBR_ENTRIES]; |
| 630 | + int tos; | |
| 630 | 631 | int lbr_callstack_users; |
| 631 | 632 | int lbr_stack_state; |
| 632 | 633 | }; |
| @@ -232,7 +232,7 @@ static struct event_constraint intel_hsw_event_constraints[] = { · 231 unmodified lines | ||
| 232 | 232 | FIXED_EVENT_CONSTRAINT(0x00c0, 0), /* INST_RETIRED.ANY */ |
| 233 | 233 | FIXED_EVENT_CONSTRAINT(0x003c, 1), /* CPU_CLK_UNHALTED.CORE */ |
| 234 | 234 | FIXED_EVENT_CONSTRAINT(0x0300, 2), /* CPU_CLK_UNHALTED.REF */ |
| 235 | − INTEL_EVENT_CONSTRAINT(0x48, 0x4), /* L1D_PEND_MISS.* */ | |
| 235 | + INTEL_UEVENT_CONSTRAINT(0x148, 0x4), /* L1D_PEND_MISS.PENDING */ | |
| 236 | 236 | INTEL_UEVENT_CONSTRAINT(0x01c0, 0x2), /* INST_RETIRED.PREC_DIST */ |
| 237 | 237 | INTEL_EVENT_CONSTRAINT(0xcd, 0x8), /* MEM_TRANS_RETIRED.LOAD_LATENCY */ |
| 238 | 238 | /* CYCLE_ACTIVITY.CYCLES_L1D_PENDING */ |
| @@ -298,7 +298,7 @@ static bool __match_event(struct perf_event *a, struct perf_event *b) · 297 unmodified lines | ||
| 298 | 298 | static inline struct perf_cgroup *event_to_cgroup(struct perf_event *event) |
| 299 | 299 | { |
| 300 | 300 | if (event->attach_state & PERF_ATTACH_TASK) |
| 301 | − return perf_cgroup_from_task(event->hw.target); | |
| 301 | + return perf_cgroup_from_task(event->hw.target, event->ctx); | |
| 302 | 302 | |
| 303 | 303 | return event->cgrp; |
| 304 | 304 | } |
| @@ -239,7 +239,7 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx) · 238 unmodified lines | ||
| 239 | 239 | } |
| 240 | 240 | |
| 241 | 241 | mask = x86_pmu.lbr_nr - 1; |
| 242 | − tos = intel_pmu_lbr_tos(); | |
| 242 | + tos = task_ctx->tos; | |
| 243 | 243 | for (i = 0; i < tos; i++) { |
| 244 | 244 | lbr_idx = (tos - i) & mask; |
| 245 | 245 | wrmsrl(x86_pmu.lbr_from + lbr_idx, task_ctx->lbr_from[i]); |
| @@ -247,6 +247,7 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx) · 1 unmodified line | ||
| 247 | 247 | if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_INFO) |
| 248 | 248 | wrmsrl(MSR_LBR_INFO_0 + lbr_idx, task_ctx->lbr_info[i]); |
| 249 | 249 | } |
| 250 | + wrmsrl(x86_pmu.lbr_tos, tos); | |
| 250 | 251 | task_ctx->lbr_stack_state = LBR_NONE; |
| 251 | 252 | } |
| 252 | 253 | |
| @@ -270,6 +271,7 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx) · 17 unmodified lines | ||
| 270 | 271 | if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_INFO) |
| 271 | 272 | rdmsrl(MSR_LBR_INFO_0 + lbr_idx, task_ctx->lbr_info[i]); |
| 272 | 273 | } |
| 274 | + task_ctx->tos = tos; | |
| 273 | 275 | task_ctx->lbr_stack_state = LBR_VALID; |
| 274 | 276 | } |
| 275 | 277 | |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | /* |
| 2 | 2 | * x86 specific code for irq_work |
| 3 | 3 | * |
| 4 | − * Copyright (C) 2010 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> | |
| 4 | + * Copyright (C) 2010 Red Hat, Inc., Peter Zijlstra | |
| 5 | 5 | */ |
| 6 | 6 | |
| 7 | 7 | #include <linux/kernel.h> |
| @@ -4,10 +4,22 @@ · 3 unmodified lines | ||
| 4 | 4 | */ |
| 5 | 5 | #include <linux/platform_device.h> |
| 6 | 6 | #include <linux/module.h> |
| 7 | +#include <linux/ioport.h> | |
| 8 | + | |
| 9 | +static int found(u64 start, u64 end, void *data) | |
| 10 | +{ | |
| 11 | + return 1; | |
| 12 | +} | |
| 7 | 13 | |
| 8 | 14 | static __init int register_e820_pmem(void) |
| 9 | 15 | { |
| 16 | + char *pmem = "Persistent Memory (legacy)"; | |
| 10 | 17 | struct platform_device *pdev; |
| 18 | + int rc; | |
| 19 | + | |
| 20 | + rc = walk_iomem_res(pmem, IORESOURCE_MEM, 0, -1, NULL, found); | |
| 21 | + if (rc <= 0) | |
| 22 | + return 0; | |
| 11 | 23 | |
| 12 | 24 | /* |
| 13 | 25 | * See drivers/nvdimm/e820.c for the implementation, this is |
| @@ -1250,8 +1250,6 @@ void __init setup_arch(char **cmdline_p) · 1249 unmodified lines | ||
| 1250 | 1250 | if (efi_enabled(EFI_BOOT)) |
| 1251 | 1251 | efi_apply_memmap_quirks(); |
| 1252 | 1252 | #endif |
| 1253 | − | |
| 1254 | − microcode_init(); | |
| 1255 | 1253 | } |
| 1256 | 1254 | |
| 1257 | 1255 | #ifdef CONFIG_X86_32 |
| @@ -690,12 +690,15 @@ handle_signal(struct ksignal *ksig, struct pt_regs *regs) · 689 unmodified lines | ||
| 690 | 690 | signal_setup_done(failed, ksig, stepping); |
| 691 | 691 | } |
| 692 | 692 | |
| 693 | −#ifdef CONFIG_X86_32 | |
| 694 | −#define NR_restart_syscall __NR_restart_syscall | |
| 695 | −#else /* !CONFIG_X86_32 */ | |
| 696 | −#define NR_restart_syscall \ | |
| 697 | − test_thread_flag(TIF_IA32) ? __NR_ia32_restart_syscall : __NR_restart_syscall | |
| 698 | −#endif /* CONFIG_X86_32 */ | |
| 693 | +static inline unsigned long get_nr_restart_syscall(const struct pt_regs *regs) | |
| 694 | +{ | |
| 695 | +#if defined(CONFIG_X86_32) || !defined(CONFIG_X86_64) | |
| 696 | + return __NR_restart_syscall; | |
| 697 | +#else /* !CONFIG_X86_32 && CONFIG_X86_64 */ | |
| 698 | + return test_thread_flag(TIF_IA32) ? __NR_ia32_restart_syscall : | |
| 699 | + __NR_restart_syscall | (regs->orig_ax & __X32_SYSCALL_BIT); | |
| 700 | +#endif /* CONFIG_X86_32 || !CONFIG_X86_64 */ | |
| 701 | +} | |
| 699 | 702 | |
| 700 | 703 | /* |
| 701 | 704 | * Note that 'init' is a special process: it doesn't get signals it doesn't |
| @@ -724,7 +727,7 @@ void do_signal(struct pt_regs *regs) · 22 unmodified lines | ||
| 724 | 727 | break; |
| 725 | 728 | |
| 726 | 729 | case -ERESTART_RESTARTBLOCK: |
| 727 | − regs->ax = NR_restart_syscall; | |
| 730 | + regs->ax = get_nr_restart_syscall(regs); | |
| 728 | 731 | regs->ip -= 2; |
| 729 | 732 | break; |
| 730 | 733 | } |
| @@ -509,7 +509,7 @@ void __inquire_remote_apic(int apicid) · 508 unmodified lines | ||
| 509 | 509 | */ |
| 510 | 510 | #define UDELAY_10MS_DEFAULT 10000 |
| 511 | 511 | |
| 512 | −static unsigned int init_udelay = INT_MAX; | |
| 512 | +static unsigned int init_udelay = UINT_MAX; | |
| 513 | 513 | |
| 514 | 514 | static int __init cpu_init_udelay(char *str) |
| 515 | 515 | { |
| @@ -522,14 +522,15 @@ early_param("cpu_init_udelay", cpu_init_udelay); · 6 unmodified lines | ||
| 522 | 522 | static void __init smp_quirk_init_udelay(void) |
| 523 | 523 | { |
| 524 | 524 | /* if cmdline changed it from default, leave it alone */ |
| 525 | − if (init_udelay != INT_MAX) | |
| 525 | + if (init_udelay != UINT_MAX) | |
| 526 | 526 | return; |
| 527 | 527 | |
| 528 | 528 | /* if modern processor, use no delay */ |
| 529 | 529 | if (((boot_cpu_data.x86_vendor == X86_VENDOR_INTEL) && (boot_cpu_data.x86 == 6)) || |
| 530 | − ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD) && (boot_cpu_data.x86 >= 0xF))) | |
| 530 | + ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD) && (boot_cpu_data.x86 >= 0xF))) { | |
| 531 | 531 | init_udelay = 0; |
| 532 | − | |
| 532 | + return; | |
| 533 | + } | |
| 533 | 534 | /* else, use legacy delay */ |
| 534 | 535 | init_udelay = UDELAY_10MS_DEFAULT; |
| 535 | 536 | } |
| @@ -101,19 +101,19 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs, · 100 unmodified lines | ||
| 101 | 101 | switch (type) { |
| 102 | 102 | case REG_TYPE_RM: |
| 103 | 103 | regno = X86_MODRM_RM(insn->modrm.value); |
| 104 | − if (X86_REX_B(insn->rex_prefix.value) == 1) | |
| 104 | + if (X86_REX_B(insn->rex_prefix.value)) | |
| 105 | 105 | regno += 8; |
| 106 | 106 | break; |
| 107 | 107 | |
| 108 | 108 | case REG_TYPE_INDEX: |
| 109 | 109 | regno = X86_SIB_INDEX(insn->sib.value); |
| 110 | − if (X86_REX_X(insn->rex_prefix.value) == 1) | |
| 110 | + if (X86_REX_X(insn->rex_prefix.value)) | |
| 111 | 111 | regno += 8; |
| 112 | 112 | break; |
| 113 | 113 | |
| 114 | 114 | case REG_TYPE_BASE: |
| 115 | 115 | regno = X86_SIB_BASE(insn->sib.value); |
| 116 | − if (X86_REX_B(insn->rex_prefix.value) == 1) | |
| 116 | + if (X86_REX_B(insn->rex_prefix.value)) | |
| 117 | 117 | regno += 8; |
| 118 | 118 | break; |
| 119 | 119 | |
| @@ -50,18 +50,9 @@ void x86_pci_root_bus_resources(int bus, struct list_head *resources) · 49 unmodified lines | ||
| 50 | 50 | if (!found) |
| 51 | 51 | pci_add_resource(resources, &info->busn); |
| 52 | 52 | |
| 53 | − list_for_each_entry(root_res, &info->resources, list) { | |
| 54 | − struct resource *res; | |
| 55 | − struct resource *root; | |
| 53 | + list_for_each_entry(root_res, &info->resources, list) | |
| 54 | + pci_add_resource(resources, &root_res->res); | |
| 56 | 55 | |
| 57 | − res = &root_res->res; | |
| 58 | − pci_add_resource(resources, res); | |
| 59 | − if (res->flags & IORESOURCE_IO) | |
| 60 | − root = &ioport_resource; | |
| 61 | − else | |
| 62 | − root = &iomem_resource; | |
| 63 | − insert_resource(root, res); | |
| 64 | − } | |
| 65 | 56 | return; |
| 66 | 57 | |
| 67 | 58 | default_resources: |
| @@ -211,7 +211,7 @@ static int copy_sc_from_user(struct pt_regs *regs, · 210 unmodified lines | ||
| 211 | 211 | if (err) |
| 212 | 212 | return 1; |
| 213 | 213 | |
| 214 | − err = convert_fxsr_from_user(&fpx, sc.fpstate); | |
| 214 | + err = convert_fxsr_from_user(&fpx, (void *)sc.fpstate); | |
| 215 | 215 | if (err) |
| 216 | 216 | return 1; |
| 217 | 217 | |
| @@ -227,7 +227,7 @@ static int copy_sc_from_user(struct pt_regs *regs, · 9 unmodified lines | ||
| 227 | 227 | { |
| 228 | 228 | struct user_i387_struct fp; |
| 229 | 229 | |
| 230 | − err = copy_from_user(&fp, sc.fpstate, | |
| 230 | + err = copy_from_user(&fp, (void *)sc.fpstate, | |
| 231 | 231 | sizeof(struct user_i387_struct)); |
| 232 | 232 | if (err) |
| 233 | 233 | return 1; |
| @@ -291,7 +291,7 @@ static int copy_sc_to_user(struct sigcontext __user *to, · 57 unmodified lines | ||
| 291 | 291 | #endif |
| 292 | 292 | #undef PUTREG |
| 293 | 293 | sc.oldmask = mask; |
| 294 | − sc.fpstate = to_fp; | |
| 294 | + sc.fpstate = (unsigned long)to_fp; | |
| 295 | 295 | |
| 296 | 296 | err = copy_to_user(to, &sc, sizeof(struct sigcontext)); |
| 297 | 297 | if (err) |
| @@ -468,12 +468,10 @@ long sys_sigreturn(void) · 170 unmodified lines | ||
| 468 | 468 | struct sigframe __user *frame = (struct sigframe __user *)(sp - 8); |
| 469 | 469 | sigset_t set; |
| 470 | 470 | struct sigcontext __user *sc = &frame->sc; |
| 471 | − unsigned long __user *oldmask = &sc->oldmask; | |
| 472 | − unsigned long __user *extramask = frame->extramask; | |
| 473 | 471 | int sig_size = (_NSIG_WORDS - 1) * sizeof(unsigned long); |
| 474 | 472 | |
| 475 | − if (copy_from_user(&set.sig[0], oldmask, sizeof(set.sig[0])) || | |
| 476 | − copy_from_user(&set.sig[1], extramask, sig_size)) | |
| 473 | + if (copy_from_user(&set.sig[0], (void *)sc->oldmask, sizeof(set.sig[0])) || | |
| 474 | + copy_from_user(&set.sig[1], frame->extramask, sig_size)) | |
| 477 | 475 | goto segfault; |
| 478 | 476 | |
| 479 | 477 | set_current_blocked(&set); |
| @@ -505,6 +503,7 @@ int setup_signal_stack_si(unsigned long stack_top, struct ksignal *ksig, · 25 unmodified lines | ||
| 505 | 503 | { |
| 506 | 504 | struct rt_sigframe __user *frame; |
| 507 | 505 | int err = 0, sig = ksig->sig; |
| 506 | + unsigned long fp_to; | |
| 508 | 507 | |
| 509 | 508 | frame = (struct rt_sigframe __user *) |
| 510 | 509 | round_down(stack_top - sizeof(struct rt_sigframe), 16); |
| @@ -526,7 +525,10 @@ int setup_signal_stack_si(unsigned long stack_top, struct ksignal *ksig, · 15 unmodified lines | ||
| 526 | 525 | err |= __save_altstack(&frame->uc.uc_stack, PT_REGS_SP(regs)); |
| 527 | 526 | err |= copy_sc_to_user(&frame->uc.uc_mcontext, &frame->fpstate, regs, |
| 528 | 527 | set->sig[0]); |
| 529 | − err |= __put_user(&frame->fpstate, &frame->uc.uc_mcontext.fpstate); | |
| 528 | + | |
| 529 | + fp_to = (unsigned long)&frame->fpstate; | |
| 530 | + | |
| 531 | + err |= __put_user(fp_to, &frame->uc.uc_mcontext.fpstate); | |
| 530 | 532 | if (sizeof(*set) == 16) { |
| 531 | 533 | err |= __put_user(set->sig[0], &frame->uc.uc_sigmask.sig[0]); |
| 532 | 534 | err |= __put_user(set->sig[1], &frame->uc.uc_sigmask.sig[1]); |
| @@ -1127,15 +1127,15 @@ void blkcg_exit_queue(struct request_queue *q) · 1126 unmodified lines | ||
| 1127 | 1127 | * of the main cic data structures. For now we allow a task to change |
| 1128 | 1128 | * its cgroup only if it's the only owner of its ioc. |
| 1129 | 1129 | */ |
| 1130 | −static int blkcg_can_attach(struct cgroup_subsys_state *css, | |
| 1131 | − struct cgroup_taskset *tset) | |
| 1130 | +static int blkcg_can_attach(struct cgroup_taskset *tset) | |
| 1132 | 1131 | { |
| 1133 | 1132 | struct task_struct *task; |
| 1133 | + struct cgroup_subsys_state *dst_css; | |
| 1134 | 1134 | struct io_context *ioc; |
| 1135 | 1135 | int ret = 0; |
| 1136 | 1136 | |
| 1137 | 1137 | /* task_lock() is needed to avoid races with exit_io_context() */ |
| 1138 | − cgroup_taskset_for_each(task, tset) { | |
| 1138 | + cgroup_taskset_for_each(task, dst_css, tset) { | |
| 1139 | 1139 | task_lock(task); |
| 1140 | 1140 | ioc = task->io_context; |
| 1141 | 1141 | if (ioc && atomic_read(&ioc->nr_tasks) > 1) |
| @@ -2114,7 +2114,8 @@ blk_qc_t submit_bio(int rw, struct bio *bio) · 2113 unmodified lines | ||
| 2114 | 2114 | EXPORT_SYMBOL(submit_bio); |
| 2115 | 2115 | |
| 2116 | 2116 | /** |
| 2117 | − * blk_rq_check_limits - Helper function to check a request for the queue limit | |
| 2117 | + * blk_cloned_rq_check_limits - Helper function to check a cloned request | |
| 2118 | + * for new the queue limits | |
| 2118 | 2119 | * @q: the queue |
| 2119 | 2120 | * @rq: the request being checked |
| 2120 | 2121 | * |
| @@ -2125,20 +2126,13 @@ EXPORT_SYMBOL(submit_bio); · 4 unmodified lines | ||
| 2125 | 2126 | * after it is inserted to @q, it should be checked against @q before |
| 2126 | 2127 | * the insertion using this generic function. |
| 2127 | 2128 | * |
| 2128 | − * This function should also be useful for request stacking drivers | |
| 2129 | − * in some cases below, so export this function. | |
| 2130 | 2129 | * Request stacking drivers like request-based dm may change the queue |
| 2131 | − * limits while requests are in the queue (e.g. dm's table swapping). | |
| 2132 | − * Such request stacking drivers should check those requests against | |
| 2133 | − * the new queue limits again when they dispatch those requests, | |
| 2134 | − * although such checkings are also done against the old queue limits | |
| 2135 | − * when submitting requests. | |
| 2130 | + * limits when retrying requests on other queues. Those requests need | |
| 2131 | + * to be checked against the new queue limits again during dispatch. | |
| 2136 | 2132 | */ |
| 2137 | −int blk_rq_check_limits(struct request_queue *q, struct request *rq) | |
| 2133 | +static int blk_cloned_rq_check_limits(struct request_queue *q, | |
| 2134 | + struct request *rq) | |
| 2138 | 2135 | { |
| 2139 | − if (!rq_mergeable(rq)) | |
| 2140 | − return 0; | |
| 2141 | − | |
| 2142 | 2136 | if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->cmd_flags)) { |
| 2143 | 2137 | printk(KERN_ERR "%s: over max size limit.\n", __func__); |
| 2144 | 2138 | return -EIO; |
| @@ -2158,7 +2152,6 @@ int blk_rq_check_limits(struct request_queue *q, struct request *rq) · 13 unmodified lines | ||
| 2158 | 2152 | |
| 2159 | 2153 | return 0; |
| 2160 | 2154 | } |
| 2161 | −EXPORT_SYMBOL_GPL(blk_rq_check_limits); | |
| 2162 | 2155 | |
| 2163 | 2156 | /** |
| 2164 | 2157 | * blk_insert_cloned_request - Helper for stacking drivers to submit a request |
| @@ -2170,7 +2163,7 @@ int blk_insert_cloned_request(struct request_queue *q, struct request *rq) · 5 unmodified lines | ||
| 2170 | 2163 | unsigned long flags; |
| 2171 | 2164 | int where = ELEVATOR_INSERT_BACK; |
| 2172 | 2165 | |
| 2173 | − if (blk_rq_check_limits(q, rq)) | |
| 2166 | + if (blk_cloned_rq_check_limits(q, rq)) | |
| 2174 | 2167 | return -EIO; |
| 2175 | 2168 | |
| 2176 | 2169 | if (rq->rq_disk && |
| @@ -3412,6 +3405,9 @@ int blk_pre_runtime_suspend(struct request_queue *q) · 1235 unmodified lines | ||
| 3412 | 3405 | { |
| 3413 | 3406 | int ret = 0; |
| 3414 | 3407 | |
| 3408 | + if (!q->dev) | |
| 3409 | + return ret; | |
| 3410 | + | |
| 3415 | 3411 | spin_lock_irq(q->queue_lock); |
| 3416 | 3412 | if (q->nr_pending) { |
| 3417 | 3413 | ret = -EBUSY; |
| @@ -3439,6 +3435,9 @@ EXPORT_SYMBOL(blk_pre_runtime_suspend); · 21 unmodified lines | ||
| 3439 | 3435 | */ |
| 3440 | 3436 | void blk_post_runtime_suspend(struct request_queue *q, int err) |
| 3441 | 3437 | { |
| 3438 | + if (!q->dev) | |
| 3439 | + return; | |
| 3440 | + | |
| 3442 | 3441 | spin_lock_irq(q->queue_lock); |
| 3443 | 3442 | if (!err) { |
| 3444 | 3443 | q->rpm_status = RPM_SUSPENDED; |
| @@ -3463,6 +3462,9 @@ EXPORT_SYMBOL(blk_post_runtime_suspend); · 18 unmodified lines | ||
| 3463 | 3462 | */ |
| 3464 | 3463 | void blk_pre_runtime_resume(struct request_queue *q) |
| 3465 | 3464 | { |
| 3465 | + if (!q->dev) | |
| 3466 | + return; | |
| 3467 | + | |
| 3466 | 3468 | spin_lock_irq(q->queue_lock); |
| 3467 | 3469 | q->rpm_status = RPM_RESUMING; |
| 3468 | 3470 | spin_unlock_irq(q->queue_lock); |
| @@ -3485,6 +3487,9 @@ EXPORT_SYMBOL(blk_pre_runtime_resume); · 16 unmodified lines | ||
| 3485 | 3487 | */ |
| 3486 | 3488 | void blk_post_runtime_resume(struct request_queue *q, int err) |
| 3487 | 3489 | { |
| 3490 | + if (!q->dev) | |
| 3491 | + return; | |
| 3492 | + | |
| 3488 | 3493 | spin_lock_irq(q->queue_lock); |
| 3489 | 3494 | if (!err) { |
| 3490 | 3495 | q->rpm_status = RPM_ACTIVE; |
| @@ -76,6 +76,9 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, · 75 unmodified lines | ||
| 76 | 76 | struct bio_vec bv, bvprv, *bvprvp = NULL; |
| 77 | 77 | struct bvec_iter iter; |
| 78 | 78 | unsigned seg_size = 0, nsegs = 0, sectors = 0; |
| 79 | + unsigned front_seg_size = bio->bi_seg_front_size; | |
| 80 | + bool do_split = true; | |
| 81 | + struct bio *new = NULL; | |
| 79 | 82 | |
| 80 | 83 | bio_for_each_segment(bv, bio, iter) { |
| 81 | 84 | if (sectors + (bv.bv_len >> 9) > queue_max_sectors(q)) |
| @@ -98,8 +101,11 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, · 16 unmodified lines | ||
| 98 | 101 | |
| 99 | 102 | seg_size += bv.bv_len; |
| 100 | 103 | bvprv = bv; |
| 101 | − bvprvp = &bv; | |
| 104 | + bvprvp = &bvprv; | |
| 102 | 105 | sectors += bv.bv_len >> 9; |
| 106 | + | |
| 107 | + if (nsegs == 1 && seg_size > front_seg_size) | |
| 108 | + front_seg_size = seg_size; | |
| 103 | 109 | continue; |
| 104 | 110 | } |
| 105 | 111 | new_segment: |
| @@ -108,16 +114,29 @@ new_segment: · 2 unmodified lines | ||
| 108 | 114 | |
| 109 | 115 | nsegs++; |
| 110 | 116 | bvprv = bv; |
| 111 | − bvprvp = &bv; | |
| 117 | + bvprvp = &bvprv; | |
| 112 | 118 | seg_size = bv.bv_len; |
| 113 | 119 | sectors += bv.bv_len >> 9; |
| 120 | + | |
| 121 | + if (nsegs == 1 && seg_size > front_seg_size) | |
| 122 | + front_seg_size = seg_size; | |
| 114 | 123 | } |
| 115 | 124 | |
| 116 | − *segs = nsegs; | |
| 117 | − return NULL; | |
| 125 | + do_split = false; | |
| 118 | 126 | split: |
| 119 | 127 | *segs = nsegs; |
| 120 | − return bio_split(bio, sectors, GFP_NOIO, bs); | |
| 128 | + | |
| 129 | + if (do_split) { | |
| 130 | + new = bio_split(bio, sectors, GFP_NOIO, bs); | |
| 131 | + if (new) | |
| 132 | + bio = new; | |
| 133 | + } | |
| 134 | + | |
| 135 | + bio->bi_seg_front_size = front_seg_size; | |
| 136 | + if (seg_size > bio->bi_seg_back_size) | |
| 137 | + bio->bi_seg_back_size = seg_size; | |
| 138 | + | |
| 139 | + return do_split ? new : NULL; | |
| 121 | 140 | } |
| 122 | 141 | |
| 123 | 142 | void blk_queue_split(struct request_queue *q, struct bio **bio, |
| @@ -412,6 +431,12 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq, · 288 unmodified lines | ||
| 412 | 431 | if (sg) |
| 413 | 432 | sg_mark_end(sg); |
| 414 | 433 | |
| 434 | + /* | |
| 435 | + * Something must have been wrong if the figured number of | |
| 436 | + * segment is bigger than number of req's physical segments | |
| 437 | + */ | |
| 438 | + WARN_ON(nsegs > rq->nr_phys_segments); | |
| 439 | + | |
| 415 | 440 | return nsegs; |
| 416 | 441 | } |
| 417 | 442 | EXPORT_SYMBOL(blk_rq_map_sg); |
| @@ -1291,15 +1291,16 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio) · 1290 unmodified lines | ||
| 1291 | 1291 | blk_mq_bio_to_request(rq, bio); |
| 1292 | 1292 | |
| 1293 | 1293 | /* |
| 1294 | − * we do limited pluging. If bio can be merged, do merge. | |
| 1294 | + * We do limited pluging. If the bio can be merged, do that. | |
| 1295 | 1295 | * Otherwise the existing request in the plug list will be |
| 1296 | 1296 | * issued. So the plug list will have one request at most |
| 1297 | 1297 | */ |
| 1298 | 1298 | if (plug) { |
| 1299 | 1299 | /* |
| 1300 | 1300 | * The plug list might get flushed before this. If that |
| 1301 | − * happens, same_queue_rq is invalid and plug list is empty | |
| 1302 | − **/ | |
| 1301 | + * happens, same_queue_rq is invalid and plug list is | |
| 1302 | + * empty | |
| 1303 | + */ | |
| 1303 | 1304 | if (same_queue_rq && !list_empty(&plug->mq_list)) { |
| 1304 | 1305 | old_rq = same_queue_rq; |
| 1305 | 1306 | list_del_init(&old_rq->queuelist); |
| @@ -1380,12 +1381,15 @@ static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio) · 74 unmodified lines | ||
| 1380 | 1381 | blk_mq_bio_to_request(rq, bio); |
| 1381 | 1382 | if (!request_count) |
| 1382 | 1383 | trace_block_plug(q); |
| 1383 | − else if (request_count >= BLK_MAX_REQUEST_COUNT) { | |
| 1384 | + | |
| 1385 | + blk_mq_put_ctx(data.ctx); | |
| 1386 | + | |
| 1387 | + if (request_count >= BLK_MAX_REQUEST_COUNT) { | |
| 1384 | 1388 | blk_flush_plug_list(plug, false); |
| 1385 | 1389 | trace_block_plug(q); |
| 1386 | 1390 | } |
| 1391 | + | |
| 1387 | 1392 | list_add_tail(&rq->queuelist, &plug->mq_list); |
| 1388 | − blk_mq_put_ctx(data.ctx); | |
| 1389 | 1393 | return cookie; |
| 1390 | 1394 | } |
| 1391 | 1395 | |
| @@ -91,7 +91,8 @@ void blk_set_default_limits(struct queue_limits *lim) · 90 unmodified lines | ||
| 91 | 91 | lim->seg_boundary_mask = BLK_SEG_BOUNDARY_MASK; |
| 92 | 92 | lim->virt_boundary_mask = 0; |
| 93 | 93 | lim->max_segment_size = BLK_MAX_SEGMENT_SIZE; |
| 94 | − lim->max_sectors = lim->max_hw_sectors = BLK_SAFE_MAX_SECTORS; | |
| 94 | + lim->max_sectors = lim->max_dev_sectors = lim->max_hw_sectors = | |
| 95 | + BLK_SAFE_MAX_SECTORS; | |
| 95 | 96 | lim->chunk_sectors = 0; |
| 96 | 97 | lim->max_write_same_sectors = 0; |
| 97 | 98 | lim->max_discard_sectors = 0; |
| @@ -127,6 +128,7 @@ void blk_set_stacking_limits(struct queue_limits *lim) · 29 unmodified lines | ||
| 127 | 128 | lim->max_hw_sectors = UINT_MAX; |
| 128 | 129 | lim->max_segment_size = UINT_MAX; |
| 129 | 130 | lim->max_sectors = UINT_MAX; |
| 131 | + lim->max_dev_sectors = UINT_MAX; | |
| 130 | 132 | lim->max_write_same_sectors = UINT_MAX; |
| 131 | 133 | } |
| 132 | 134 | EXPORT_SYMBOL(blk_set_stacking_limits); |
| @@ -214,8 +216,8 @@ void blk_queue_bounce_limit(struct request_queue *q, u64 max_addr) · 81 unmodified lines | ||
| 214 | 216 | EXPORT_SYMBOL(blk_queue_bounce_limit); |
| 215 | 217 | |
| 216 | 218 | /** |
| 217 | − * blk_limits_max_hw_sectors - set hard and soft limit of max sectors for request | |
| 218 | − * @limits: the queue limits | |
| 219 | + * blk_queue_max_hw_sectors - set max sectors for a request for this queue | |
| 220 | + * @q: the request queue for the device | |
| 219 | 221 | * @max_hw_sectors: max hardware sectors in the usual 512b unit |
| 220 | 222 | * |
| 221 | 223 | * Description: |
| @@ -224,13 +226,19 @@ EXPORT_SYMBOL(blk_queue_bounce_limit); · 2 unmodified lines | ||
| 224 | 226 | * the device driver based upon the capabilities of the I/O |
| 225 | 227 | * controller. |
| 226 | 228 | * |
| 229 | + * max_dev_sectors is a hard limit imposed by the storage device for | |
| 230 | + * READ/WRITE requests. It is set by the disk driver. | |
| 231 | + * | |
| 227 | 232 | * max_sectors is a soft limit imposed by the block layer for |
| 228 | 233 | * filesystem type requests. This value can be overridden on a |
| 229 | 234 | * per-device basis in /sys/block/<device>/queue/max_sectors_kb. |
| 230 | 235 | * The soft limit can not exceed max_hw_sectors. |
| 231 | 236 | **/ |
| 232 | −void blk_limits_max_hw_sectors(struct queue_limits *limits, unsigned int max_hw_sectors) | |
| 237 | +void blk_queue_max_hw_sectors(struct request_queue *q, unsigned int max_hw_sectors) | |
| 233 | 238 | { |
| 239 | + struct queue_limits *limits = &q->limits; | |
| 240 | + unsigned int max_sectors; | |
| 241 | + | |
| 234 | 242 | if ((max_hw_sectors << 9) < PAGE_CACHE_SIZE) { |
| 235 | 243 | max_hw_sectors = 1 << (PAGE_CACHE_SHIFT - 9); |
| 236 | 244 | printk(KERN_INFO "%s: set to minimum %d\n", |
| @@ -238,22 +246,9 @@ void blk_limits_max_hw_sectors(struct queue_limits *limits, unsigned int max_hw_ · 1 unmodified line | ||
| 238 | 246 | } |
| 239 | 247 | |
| 240 | 248 | limits->max_hw_sectors = max_hw_sectors; |
| 241 | − limits->max_sectors = min_t(unsigned int, max_hw_sectors, | |
| 242 | − BLK_DEF_MAX_SECTORS); | |
| 243 | −} | |
| 244 | −EXPORT_SYMBOL(blk_limits_max_hw_sectors); | |
| 245 | − | |
| 246 | −/** | |
| 247 | − * blk_queue_max_hw_sectors - set max sectors for a request for this queue | |
| 248 | − * @q: the request queue for the device | |
| 249 | − * @max_hw_sectors: max hardware sectors in the usual 512b unit | |
| 250 | − * | |
| 251 | − * Description: | |
| 252 | − * See description for blk_limits_max_hw_sectors(). | |
| 253 | − **/ | |
| 254 | −void blk_queue_max_hw_sectors(struct request_queue *q, unsigned int max_hw_sectors) | |
| 255 | −{ | |
| 256 | − blk_limits_max_hw_sectors(&q->limits, max_hw_sectors); | |
| 249 | + max_sectors = min_not_zero(max_hw_sectors, limits->max_dev_sectors); | |
| 250 | + max_sectors = min_t(unsigned int, max_sectors, BLK_DEF_MAX_SECTORS); | |
| 251 | + limits->max_sectors = max_sectors; | |
| 257 | 252 | } |
| 258 | 253 | EXPORT_SYMBOL(blk_queue_max_hw_sectors); |
| 259 | 254 | |
| @@ -527,6 +522,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, · 267 unmodified lines | ||
| 527 | 522 | |
| 528 | 523 | t->max_sectors = min_not_zero(t->max_sectors, b->max_sectors); |
| 529 | 524 | t->max_hw_sectors = min_not_zero(t->max_hw_sectors, b->max_hw_sectors); |
| 525 | + t->max_dev_sectors = min_not_zero(t->max_dev_sectors, b->max_dev_sectors); | |
| 530 | 526 | t->max_write_same_sectors = min(t->max_write_same_sectors, |
| 531 | 527 | b->max_write_same_sectors); |
| 532 | 528 | t->bounce_pfn = min_not_zero(t->bounce_pfn, b->bounce_pfn); |
| @@ -205,6 +205,9 @@ queue_max_sectors_store(struct request_queue *q, const char *page, size_t count) · 204 unmodified lines | ||
| 205 | 205 | if (ret < 0) |
| 206 | 206 | return ret; |
| 207 | 207 | |
| 208 | + max_hw_sectors_kb = min_not_zero(max_hw_sectors_kb, (unsigned long) | |
| 209 | + q->limits.max_dev_sectors >> 1); | |
| 210 | + | |
| 208 | 211 | if (max_sectors_kb > max_hw_sectors_kb || max_sectors_kb < page_kb) |
| 209 | 212 | return -EINVAL; |
| 210 | 213 | |
| @@ -158,11 +158,13 @@ void blk_abort_request(struct request *req) · 157 unmodified lines | ||
| 158 | 158 | { |
| 159 | 159 | if (blk_mark_rq_complete(req)) |
| 160 | 160 | return; |
| 161 | − blk_delete_timer(req); | |
| 162 | − if (req->q->mq_ops) | |
| 161 | + | |
| 162 | + if (req->q->mq_ops) { | |
| 163 | 163 | blk_mq_rq_timed_out(req, false); |
| 164 | − else | |
| 164 | + } else { | |
| 165 | + blk_delete_timer(req); | |
| 165 | 166 | blk_rq_timed_out(req); |
| 167 | + } | |
| 166 | 168 | } |
| 167 | 169 | EXPORT_SYMBOL_GPL(blk_abort_request); |
| 168 | 170 | |
| @@ -21,10 +21,10 @@ static void noop_merged_requests(struct request_queue *q, struct request *rq, · 20 unmodified lines | ||
| 21 | 21 | static int noop_dispatch(struct request_queue *q, int force) |
| 22 | 22 | { |
| 23 | 23 | struct noop_data *nd = q->elevator->elevator_data; |
| 24 | + struct request *rq; | |
| 24 | 25 | |
| 25 | − if (!list_empty(&nd->queue)) { | |
| 26 | − struct request *rq; | |
| 27 | − rq = list_entry(nd->queue.next, struct request, queuelist); | |
| 26 | + rq = list_first_entry_or_null(&nd->queue, struct request, queuelist); | |
| 27 | + if (rq) { | |
| 28 | 28 | list_del_init(&rq->queuelist); |
| 29 | 29 | elv_dispatch_sort(q, rq); |
| 30 | 30 | return 1; |
| @@ -46,7 +46,7 @@ noop_former_request(struct request_queue *q, struct request *rq) · 15 unmodified lines | ||
| 46 | 46 | |
| 47 | 47 | if (rq->queuelist.prev == &nd->queue) |
| 48 | 48 | return NULL; |
| 49 | − return list_entry(rq->queuelist.prev, struct request, queuelist); | |
| 49 | + return list_prev_entry(rq, queuelist); | |
| 50 | 50 | } |
| 51 | 51 | |
| 52 | 52 | static struct request * |
| @@ -56,7 +56,7 @@ noop_latter_request(struct request_queue *q, struct request *rq) · 3 unmodified lines | ||
| 56 | 56 | |
| 57 | 57 | if (rq->queuelist.next == &nd->queue) |
| 58 | 58 | return NULL; |
| 59 | − return list_entry(rq->queuelist.next, struct request, queuelist); | |
| 59 | + return list_next_entry(rq, queuelist); | |
| 60 | 60 | } |
| 61 | 61 | |
| 62 | 62 | static int noop_init_queue(struct request_queue *q, struct elevator_type *e) |
| @@ -397,7 +397,7 @@ static int drop_partitions(struct gendisk *disk, struct block_device *bdev) · 396 unmodified lines | ||
| 397 | 397 | struct hd_struct *part; |
| 398 | 398 | int res; |
| 399 | 399 | |
| 400 | − if (bdev->bd_part_count) | |
| 400 | + if (bdev->bd_part_count || bdev->bd_super) | |
| 401 | 401 | return -EBUSY; |
| 402 | 402 | res = invalidate_partition(disk, 0); |
| 403 | 403 | if (res) |
| @@ -32,7 +32,7 @@ int mac_partition(struct parsed_partitions *state) · 31 unmodified lines | ||
| 32 | 32 | Sector sect; |
| 33 | 33 | unsigned char *data; |
| 34 | 34 | int slot, blocks_in_map; |
| 35 | − unsigned secsize; | |
| 35 | + unsigned secsize, datasize, partoffset; | |
| 36 | 36 | #ifdef CONFIG_PPC_PMAC |
| 37 | 37 | int found_root = 0; |
| 38 | 38 | int found_root_goodness = 0; |
| @@ -50,10 +50,14 @@ int mac_partition(struct parsed_partitions *state) · 11 unmodified lines | ||
| 50 | 50 | } |
| 51 | 51 | secsize = be16_to_cpu(md->block_size); |
| 52 | 52 | put_dev_sector(sect); |
| 53 | − data = read_part_sector(state, secsize/512, §); | |
| 53 | + datasize = round_down(secsize, 512); | |
| 54 | + data = read_part_sector(state, datasize / 512, §); | |
| 54 | 55 | if (!data) |
| 55 | 56 | return -1; |
| 56 | − part = (struct mac_partition *) (data + secsize%512); | |
| 57 | + partoffset = secsize % 512; | |
| 58 | + if (partoffset + sizeof(*part) > datasize) | |
| 59 | + return -1; | |
| 60 | + part = (struct mac_partition *) (data + partoffset); | |
| 57 | 61 | if (be16_to_cpu(part->signature) != MAC_PARTITION_MAGIC) { |
| 58 | 62 | put_dev_sector(sect); |
| 59 | 63 | return 0; /* not a MacOS disk */ |
| @@ -125,7 +125,7 @@ static int aead_wait_for_data(struct sock *sk, unsigned flags) · 124 unmodified lines | ||
| 125 | 125 | if (flags & MSG_DONTWAIT) |
| 126 | 126 | return -EAGAIN; |
| 127 | 127 | |
| 128 | − set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags); | |
| 128 | + sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); | |
| 129 | 129 | |
| 130 | 130 | for (;;) { |
| 131 | 131 | if (signal_pending(current)) |
| @@ -139,7 +139,7 @@ static int aead_wait_for_data(struct sock *sk, unsigned flags) · 7 unmodified lines | ||
| 139 | 139 | } |
| 140 | 140 | finish_wait(sk_sleep(sk), &wait); |
| 141 | 141 | |
| 142 | − clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags); | |
| 142 | + sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); | |
| 143 | 143 | |
| 144 | 144 | return err; |
| 145 | 145 | } |
| @@ -212,7 +212,7 @@ static int skcipher_wait_for_wmem(struct sock *sk, unsigned flags) · 211 unmodified lines | ||
| 212 | 212 | if (flags & MSG_DONTWAIT) |
| 213 | 213 | return -EAGAIN; |
| 214 | 214 | |
| 215 | − set_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags); | |
| 215 | + sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk); | |
| 216 | 216 | |
| 217 | 217 | for (;;) { |
| 218 | 218 | if (signal_pending(current)) |
| @@ -258,7 +258,7 @@ static int skcipher_wait_for_data(struct sock *sk, unsigned flags) · 39 unmodified lines | ||
| 258 | 258 | return -EAGAIN; |
| 259 | 259 | } |
| 260 | 260 | |
| 261 | − set_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags); | |
| 261 | + sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); | |
| 262 | 262 | |
| 263 | 263 | for (;;) { |
| 264 | 264 | if (signal_pending(current)) |
| @@ -272,7 +272,7 @@ static int skcipher_wait_for_data(struct sock *sk, unsigned flags) · 7 unmodified lines | ||
| 272 | 272 | } |
| 273 | 273 | finish_wait(sk_sleep(sk), &wait); |
| 274 | 274 | |
| 275 | − clear_bit(SOCK_ASYNC_WAITDATA, &sk->sk_socket->flags); | |
| 275 | + sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); | |
| 276 | 276 | |
| 277 | 277 | return err; |
| 278 | 278 | } |
| @@ -63,6 +63,7 @@ obj-$(CONFIG_FB_I810) += video/fbdev/i810/ · 62 unmodified lines | ||
| 63 | 63 | obj-$(CONFIG_FB_INTEL) += video/fbdev/intelfb/ |
| 64 | 64 | |
| 65 | 65 | obj-$(CONFIG_PARPORT) += parport/ |
| 66 | +obj-$(CONFIG_NVM) += lightnvm/ | |
| 66 | 67 | obj-y += base/ block/ misc/ mfd/ nfc/ |
| 67 | 68 | obj-$(CONFIG_LIBNVDIMM) += nvdimm/ |
| 68 | 69 | obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/ |
| @@ -70,7 +71,6 @@ obj-$(CONFIG_NUBUS) += nubus/ · 1 unmodified line | ||
| 70 | 71 | obj-y += macintosh/ |
| 71 | 72 | obj-$(CONFIG_IDE) += ide/ |
| 72 | 73 | obj-$(CONFIG_SCSI) += scsi/ |
| 73 | −obj-$(CONFIG_NVM) += lightnvm/ | |
| 74 | 74 | obj-y += nvme/ |
| 75 | 75 | obj-$(CONFIG_ATA) += ata/ |
| 76 | 76 | obj-$(CONFIG_TARGET_CORE) += target/ |
| @@ -58,10 +58,10 @@ config ACPI_CCA_REQUIRED · 57 unmodified lines | ||
| 58 | 58 | bool |
| 59 | 59 | |
| 60 | 60 | config ACPI_DEBUGGER |
| 61 | − bool "In-kernel debugger (EXPERIMENTAL)" | |
| 61 | + bool "AML debugger interface (EXPERIMENTAL)" | |
| 62 | 62 | select ACPI_DEBUG |
| 63 | 63 | help |
| 64 | − Enable in-kernel debugging facilities: statistics, internal | |
| 64 | + Enable in-kernel debugging of AML facilities: statistics, internal | |
| 65 | 65 | object dump, single step control method execution. |
| 66 | 66 | This is still under development, currently enabling this only |
| 67 | 67 | results in the compilation of the ACPICA debugger files. |
| @@ -233,11 +233,12 @@ static bool add_spa(struct acpi_nfit_desc *acpi_desc, · 232 unmodified lines | ||
| 233 | 233 | struct nfit_table_prev *prev, |
| 234 | 234 | struct acpi_nfit_system_address *spa) |
| 235 | 235 | { |
| 236 | + size_t length = min_t(size_t, sizeof(*spa), spa->header.length); | |
| 236 | 237 | struct device *dev = acpi_desc->dev; |
| 237 | 238 | struct nfit_spa *nfit_spa; |
| 238 | 239 | |
| 239 | 240 | list_for_each_entry(nfit_spa, &prev->spas, list) { |
| 240 | − if (memcmp(nfit_spa->spa, spa, sizeof(*spa)) == 0) { | |
| 241 | + if (memcmp(nfit_spa->spa, spa, length) == 0) { | |
| 241 | 242 | list_move_tail(&nfit_spa->list, &acpi_desc->spas); |
| 242 | 243 | return true; |
| 243 | 244 | } |
| @@ -259,11 +260,12 @@ static bool add_memdev(struct acpi_nfit_desc *acpi_desc, · 15 unmodified lines | ||
| 259 | 260 | struct nfit_table_prev *prev, |
| 260 | 261 | struct acpi_nfit_memory_map *memdev) |
| 261 | 262 | { |
| 263 | + size_t length = min_t(size_t, sizeof(*memdev), memdev->header.length); | |
| 262 | 264 | struct device *dev = acpi_desc->dev; |
| 263 | 265 | struct nfit_memdev *nfit_memdev; |
| 264 | 266 | |
| 265 | 267 | list_for_each_entry(nfit_memdev, &prev->memdevs, list) |
| 266 | − if (memcmp(nfit_memdev->memdev, memdev, sizeof(*memdev)) == 0) { | |
| 268 | + if (memcmp(nfit_memdev->memdev, memdev, length) == 0) { | |
| 267 | 269 | list_move_tail(&nfit_memdev->list, &acpi_desc->memdevs); |
| 268 | 270 | return true; |
| 269 | 271 | } |
| @@ -284,11 +286,12 @@ static bool add_dcr(struct acpi_nfit_desc *acpi_desc, · 14 unmodified lines | ||
| 284 | 286 | struct nfit_table_prev *prev, |
| 285 | 287 | struct acpi_nfit_control_region *dcr) |
| 286 | 288 | { |
| 289 | + size_t length = min_t(size_t, sizeof(*dcr), dcr->header.length); | |
| 287 | 290 | struct device *dev = acpi_desc->dev; |
| 288 | 291 | struct nfit_dcr *nfit_dcr; |
| 289 | 292 | |
| 290 | 293 | list_for_each_entry(nfit_dcr, &prev->dcrs, list) |
| 291 | − if (memcmp(nfit_dcr->dcr, dcr, sizeof(*dcr)) == 0) { | |
| 294 | + if (memcmp(nfit_dcr->dcr, dcr, length) == 0) { | |
| 292 | 295 | list_move_tail(&nfit_dcr->list, &acpi_desc->dcrs); |
| 293 | 296 | return true; |
| 294 | 297 | } |
| @@ -308,11 +311,12 @@ static bool add_bdw(struct acpi_nfit_desc *acpi_desc, · 13 unmodified lines | ||
| 308 | 311 | struct nfit_table_prev *prev, |
| 309 | 312 | struct acpi_nfit_data_region *bdw) |
| 310 | 313 | { |
| 314 | + size_t length = min_t(size_t, sizeof(*bdw), bdw->header.length); | |
| 311 | 315 | struct device *dev = acpi_desc->dev; |
| 312 | 316 | struct nfit_bdw *nfit_bdw; |
| 313 | 317 | |
| 314 | 318 | list_for_each_entry(nfit_bdw, &prev->bdws, list) |
| 315 | − if (memcmp(nfit_bdw->bdw, bdw, sizeof(*bdw)) == 0) { | |
| 319 | + if (memcmp(nfit_bdw->bdw, bdw, length) == 0) { | |
| 316 | 320 | list_move_tail(&nfit_bdw->list, &acpi_desc->bdws); |
| 317 | 321 | return true; |
| 318 | 322 | } |
| @@ -332,11 +336,12 @@ static bool add_idt(struct acpi_nfit_desc *acpi_desc, · 13 unmodified lines | ||
| 332 | 336 | struct nfit_table_prev *prev, |
| 333 | 337 | struct acpi_nfit_interleave *idt) |
| 334 | 338 | { |
| 339 | + size_t length = min_t(size_t, sizeof(*idt), idt->header.length); | |
| 335 | 340 | struct device *dev = acpi_desc->dev; |
| 336 | 341 | struct nfit_idt *nfit_idt; |
| 337 | 342 | |
| 338 | 343 | list_for_each_entry(nfit_idt, &prev->idts, list) |
| 339 | − if (memcmp(nfit_idt->idt, idt, sizeof(*idt)) == 0) { | |
| 344 | + if (memcmp(nfit_idt->idt, idt, length) == 0) { | |
| 340 | 345 | list_move_tail(&nfit_idt->list, &acpi_desc->idts); |
| 341 | 346 | return true; |
| 342 | 347 | } |
| @@ -356,11 +361,12 @@ static bool add_flush(struct acpi_nfit_desc *acpi_desc, · 13 unmodified lines | ||
| 356 | 361 | struct nfit_table_prev *prev, |
| 357 | 362 | struct acpi_nfit_flush_address *flush) |
| 358 | 363 | { |
| 364 | + size_t length = min_t(size_t, sizeof(*flush), flush->header.length); | |
| 359 | 365 | struct device *dev = acpi_desc->dev; |
| 360 | 366 | struct nfit_flush *nfit_flush; |
| 361 | 367 | |
| 362 | 368 | list_for_each_entry(nfit_flush, &prev->flushes, list) |
| 363 | − if (memcmp(nfit_flush->flush, flush, sizeof(*flush)) == 0) { | |
| 369 | + if (memcmp(nfit_flush->flush, flush, length) == 0) { | |
| 364 | 370 | list_move_tail(&nfit_flush->list, &acpi_desc->flushes); |
| 365 | 371 | return true; |
| 366 | 372 | } |
| @@ -655,7 +661,7 @@ static ssize_t revision_show(struct device *dev, · 288 unmodified lines | ||
| 655 | 661 | struct nvdimm_bus_descriptor *nd_desc = to_nd_desc(nvdimm_bus); |
| 656 | 662 | struct acpi_nfit_desc *acpi_desc = to_acpi_desc(nd_desc); |
| 657 | 663 | |
| 658 | − return sprintf(buf, "%d\n", acpi_desc->nfit->header.revision); | |
| 664 | + return sprintf(buf, "%d\n", acpi_desc->acpi_header.revision); | |
| 659 | 665 | } |
| 660 | 666 | static DEVICE_ATTR_RO(revision); |
| 661 | 667 | |
| @@ -1652,7 +1658,6 @@ int acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, acpi_size sz) · 990 unmodified lines | ||
| 1652 | 1658 | |
| 1653 | 1659 | data = (u8 *) acpi_desc->nfit; |
| 1654 | 1660 | end = data + sz; |
| 1655 | − data += sizeof(struct acpi_table_nfit); | |
| 1656 | 1661 | while (!IS_ERR_OR_NULL(data)) |
| 1657 | 1662 | data = add_table(acpi_desc, &prev, data, end); |
| 1658 | 1663 | |
| @@ -1748,13 +1753,29 @@ static int acpi_nfit_add(struct acpi_device *adev) · 89 unmodified lines | ||
| 1748 | 1753 | return PTR_ERR(acpi_desc); |
| 1749 | 1754 | } |
| 1750 | 1755 | |
| 1751 | − acpi_desc->nfit = (struct acpi_table_nfit *) tbl; | |
| 1756 | + /* | |
| 1757 | + * Save the acpi header for later and then skip it, | |
| 1758 | + * making nfit point to the first nfit table header. | |
| 1759 | + */ | |
| 1760 | + acpi_desc->acpi_header = *tbl; | |
| 1761 | + acpi_desc->nfit = (void *) tbl + sizeof(struct acpi_table_nfit); | |
| 1762 | + sz -= sizeof(struct acpi_table_nfit); | |
| 1752 | 1763 | |
| 1753 | 1764 | /* Evaluate _FIT and override with that if present */ |
| 1754 | 1765 | status = acpi_evaluate_object(adev->handle, "_FIT", NULL, &buf); |
| 1755 | 1766 | if (ACPI_SUCCESS(status) && buf.length > 0) { |
| 1756 | − acpi_desc->nfit = (struct acpi_table_nfit *)buf.pointer; | |
| 1757 | − sz = buf.length; | |
| 1767 | + union acpi_object *obj; | |
| 1768 | + /* | |
| 1769 | + * Adjust for the acpi_object header of the _FIT | |
| 1770 | + */ | |
| 1771 | + obj = buf.pointer; | |
| 1772 | + if (obj->type == ACPI_TYPE_BUFFER) { | |
| 1773 | + acpi_desc->nfit = | |
| 1774 | + (struct acpi_nfit_header *)obj->buffer.pointer; | |
| 1775 | + sz = obj->buffer.length; | |
| 1776 | + } else | |
| 1777 | + dev_dbg(dev, "%s invalid type %d, ignoring _FIT\n", | |
| 1778 | + __func__, (int) obj->type); | |
| 1758 | 1779 | } |
| 1759 | 1780 | |
| 1760 | 1781 | rc = acpi_nfit_init(acpi_desc, sz); |
| @@ -1777,7 +1798,8 @@ static void acpi_nfit_notify(struct acpi_device *adev, u32 event) · 16 unmodified lines | ||
| 1777 | 1798 | { |
| 1778 | 1799 | struct acpi_nfit_desc *acpi_desc = dev_get_drvdata(&adev->dev); |
| 1779 | 1800 | struct acpi_buffer buf = { ACPI_ALLOCATE_BUFFER, NULL }; |
| 1780 | − struct acpi_table_nfit *nfit_saved; | |
| 1801 | + struct acpi_nfit_header *nfit_saved; | |
| 1802 | + union acpi_object *obj; | |
| 1781 | 1803 | struct device *dev = &adev->dev; |
| 1782 | 1804 | acpi_status status; |
| 1783 | 1805 | int ret; |
| @@ -1808,12 +1830,19 @@ static void acpi_nfit_notify(struct acpi_device *adev, u32 event) · 24 unmodified lines | ||
| 1808 | 1830 | } |
| 1809 | 1831 | |
| 1810 | 1832 | nfit_saved = acpi_desc->nfit; |
| 1811 | − acpi_desc->nfit = (struct acpi_table_nfit *)buf.pointer; | |
| 1812 | − ret = acpi_nfit_init(acpi_desc, buf.length); | |
| 1813 | − if (!ret) { | |
| 1814 | − /* Merge failed, restore old nfit, and exit */ | |
| 1815 | − acpi_desc->nfit = nfit_saved; | |
| 1816 | − dev_err(dev, "failed to merge updated NFIT\n"); | |
| 1833 | + obj = buf.pointer; | |
| 1834 | + if (obj->type == ACPI_TYPE_BUFFER) { | |
| 1835 | + acpi_desc->nfit = | |
| 1836 | + (struct acpi_nfit_header *)obj->buffer.pointer; | |
| 1837 | + ret = acpi_nfit_init(acpi_desc, obj->buffer.length); | |
| 1838 | + if (ret) { | |
| 1839 | + /* Merge failed, restore old nfit, and exit */ | |
| 1840 | + acpi_desc->nfit = nfit_saved; | |
| 1841 | + dev_err(dev, "failed to merge updated NFIT\n"); | |
| 1842 | + } | |
| 1843 | + } else { | |
| 1844 | + /* Bad _FIT, restore old nfit */ | |
| 1845 | + dev_err(dev, "Invalid _FIT\n"); | |
| 1817 | 1846 | } |
| 1818 | 1847 | kfree(buf.pointer); |
| 1819 | 1848 | |
| @@ -96,7 +96,8 @@ struct nfit_mem { · 95 unmodified lines | ||
| 96 | 96 | |
| 97 | 97 | struct acpi_nfit_desc { |
| 98 | 98 | struct nvdimm_bus_descriptor nd_desc; |
| 99 | − struct acpi_table_nfit *nfit; | |
| 99 | + struct acpi_table_header acpi_header; | |
| 100 | + struct acpi_nfit_header *nfit; | |
| 100 | 101 | struct mutex spa_map_mutex; |
| 101 | 102 | struct mutex init_mutex; |
| 102 | 103 | struct list_head spa_maps; |
| @@ -768,6 +768,13 @@ static void pci_acpi_root_add_resources(struct acpi_pci_root_info *info) · 767 unmodified lines | ||
| 768 | 768 | else |
| 769 | 769 | continue; |
| 770 | 770 | |
| 771 | + /* | |
| 772 | + * Some legacy x86 host bridge drivers use iomem_resource and | |
| 773 | + * ioport_resource as default resource pool, skip it. | |
| 774 | + */ | |
| 775 | + if (res == root) | |
| 776 | + continue; | |
| 777 | + | |
| 771 | 778 | conflict = insert_resource_conflict(root, res); |
| 772 | 779 | if (conflict) { |
| 773 | 780 | dev_info(&info->bridge->dev, |
| @@ -314,16 +314,6 @@ static const struct pci_device_id ahci_pci_tbl[] = { · 313 unmodified lines | ||
| 314 | 314 | { PCI_VDEVICE(INTEL, 0x1f37), board_ahci_avn }, /* Avoton RAID */ |
| 315 | 315 | { PCI_VDEVICE(INTEL, 0x1f3e), board_ahci_avn }, /* Avoton RAID */ |
| 316 | 316 | { PCI_VDEVICE(INTEL, 0x1f3f), board_ahci_avn }, /* Avoton RAID */ |
| 317 | − { PCI_VDEVICE(INTEL, 0xa182), board_ahci }, /* Lewisburg AHCI*/ | |
| 318 | − { PCI_VDEVICE(INTEL, 0xa202), board_ahci }, /* Lewisburg AHCI*/ | |
| 319 | − { PCI_VDEVICE(INTEL, 0xa184), board_ahci }, /* Lewisburg RAID*/ | |
| 320 | − { PCI_VDEVICE(INTEL, 0xa204), board_ahci }, /* Lewisburg RAID*/ | |
| 321 | − { PCI_VDEVICE(INTEL, 0xa186), board_ahci }, /* Lewisburg RAID*/ | |
| 322 | − { PCI_VDEVICE(INTEL, 0xa206), board_ahci }, /* Lewisburg RAID*/ | |
| 323 | − { PCI_VDEVICE(INTEL, 0x2822), board_ahci }, /* Lewisburg RAID*/ | |
| 324 | − { PCI_VDEVICE(INTEL, 0x2826), board_ahci }, /* Lewisburg RAID*/ | |
| 325 | − { PCI_VDEVICE(INTEL, 0xa18e), board_ahci }, /* Lewisburg RAID*/ | |
| 326 | − { PCI_VDEVICE(INTEL, 0xa20e), board_ahci }, /* Lewisburg RAID*/ | |
| 327 | 317 | { PCI_VDEVICE(INTEL, 0x2823), board_ahci }, /* Wellsburg RAID */ |
| 328 | 318 | { PCI_VDEVICE(INTEL, 0x2827), board_ahci }, /* Wellsburg RAID */ |
| 329 | 319 | { PCI_VDEVICE(INTEL, 0x8d02), board_ahci }, /* Wellsburg AHCI */ |
| @@ -350,10 +340,22 @@ static const struct pci_device_id ahci_pci_tbl[] = { · 20 unmodified lines | ||
| 350 | 340 | { PCI_VDEVICE(INTEL, 0x9d03), board_ahci }, /* Sunrise Point-LP AHCI */ |
| 351 | 341 | { PCI_VDEVICE(INTEL, 0x9d05), board_ahci }, /* Sunrise Point-LP RAID */ |
| 352 | 342 | { PCI_VDEVICE(INTEL, 0x9d07), board_ahci }, /* Sunrise Point-LP RAID */ |
| 343 | + { PCI_VDEVICE(INTEL, 0xa102), board_ahci }, /* Sunrise Point-H AHCI */ | |
| 353 | 344 | { PCI_VDEVICE(INTEL, 0xa103), board_ahci }, /* Sunrise Point-H AHCI */ |
| 354 | 345 | { PCI_VDEVICE(INTEL, 0xa105), board_ahci }, /* Sunrise Point-H RAID */ |
| 346 | + { PCI_VDEVICE(INTEL, 0xa106), board_ahci }, /* Sunrise Point-H RAID */ | |
| 355 | 347 | { PCI_VDEVICE(INTEL, 0xa107), board_ahci }, /* Sunrise Point-H RAID */ |
| 356 | 348 | { PCI_VDEVICE(INTEL, 0xa10f), board_ahci }, /* Sunrise Point-H RAID */ |
| 349 | + { PCI_VDEVICE(INTEL, 0x2822), board_ahci }, /* Lewisburg RAID*/ | |
| 350 | + { PCI_VDEVICE(INTEL, 0x2826), board_ahci }, /* Lewisburg RAID*/ | |
| 351 | + { PCI_VDEVICE(INTEL, 0xa182), board_ahci }, /* Lewisburg AHCI*/ | |
| 352 | + { PCI_VDEVICE(INTEL, 0xa184), board_ahci }, /* Lewisburg RAID*/ | |
| 353 | + { PCI_VDEVICE(INTEL, 0xa186), board_ahci }, /* Lewisburg RAID*/ | |
| 354 | + { PCI_VDEVICE(INTEL, 0xa18e), board_ahci }, /* Lewisburg RAID*/ | |
| 355 | + { PCI_VDEVICE(INTEL, 0xa202), board_ahci }, /* Lewisburg AHCI*/ | |
| 356 | + { PCI_VDEVICE(INTEL, 0xa204), board_ahci }, /* Lewisburg RAID*/ | |
| 357 | + { PCI_VDEVICE(INTEL, 0xa206), board_ahci }, /* Lewisburg RAID*/ | |
| 358 | + { PCI_VDEVICE(INTEL, 0xa20e), board_ahci }, /* Lewisburg RAID*/ | |
| 357 | 359 | |
| 358 | 360 | /* JMicron 360/1/3/5/6, match class to avoid IDE function */ |
| 359 | 361 | { PCI_VENDOR_ID_JMICRON, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, |
| @@ -62,6 +62,7 @@ static void ahci_mvebu_regret_option(struct ahci_host_priv *hpriv) · 61 unmodified lines | ||
| 62 | 62 | writel(0x80, hpriv->mmio + AHCI_VENDOR_SPECIFIC_0_DATA); |
| 63 | 63 | } |
| 64 | 64 | |
| 65 | +#ifdef CONFIG_PM_SLEEP | |
| 65 | 66 | static int ahci_mvebu_suspend(struct platform_device *pdev, pm_message_t state) |
| 66 | 67 | { |
| 67 | 68 | return ahci_platform_suspend_host(&pdev->dev); |
| @@ -81,6 +82,10 @@ static int ahci_mvebu_resume(struct platform_device *pdev) · 13 unmodified lines | ||
| 81 | 82 | |
| 82 | 83 | return ahci_platform_resume_host(&pdev->dev); |
| 83 | 84 | } |
| 85 | +#else | |
| 86 | +#define ahci_mvebu_suspend NULL | |
| 87 | +#define ahci_mvebu_resume NULL | |
| 88 | +#endif | |
| 84 | 89 | |
| 85 | 90 | static const struct ata_port_info ahci_mvebu_port_info = { |
| 86 | 91 | .flags = AHCI_FLAG_COMMON, |
| @@ -1273,6 +1273,15 @@ static int ahci_exec_polled_cmd(struct ata_port *ap, int pmp, · 1272 unmodified lines | ||
| 1273 | 1273 | ata_tf_to_fis(tf, pmp, is_cmd, fis); |
| 1274 | 1274 | ahci_fill_cmd_slot(pp, 0, cmd_fis_len | flags | (pmp << 12)); |
| 1275 | 1275 | |
| 1276 | + /* set port value for softreset of Port Multiplier */ | |
| 1277 | + if (pp->fbs_enabled && pp->fbs_last_dev != pmp) { | |
| 1278 | + tmp = readl(port_mmio + PORT_FBS); | |
| 1279 | + tmp &= ~(PORT_FBS_DEV_MASK | PORT_FBS_DEC); | |
| 1280 | + tmp |= pmp << PORT_FBS_DEV_OFFSET; | |
| 1281 | + writel(tmp, port_mmio + PORT_FBS); | |
| 1282 | + pp->fbs_last_dev = pmp; | |
| 1283 | + } | |
| 1284 | + | |
| 1276 | 1285 | /* issue & wait */ |
| 1277 | 1286 | writel(1, port_mmio + PORT_CMD_ISSUE); |
| 1278 | 1287 | |
| @@ -1505,12 +1505,20 @@ static const char *ata_err_string(unsigned int err_mask) · 1504 unmodified lines | ||
| 1505 | 1505 | unsigned int ata_read_log_page(struct ata_device *dev, u8 log, |
| 1506 | 1506 | u8 page, void *buf, unsigned int sectors) |
| 1507 | 1507 | { |
| 1508 | + unsigned long ap_flags = dev->link->ap->flags; | |
| 1508 | 1509 | struct ata_taskfile tf; |
| 1509 | 1510 | unsigned int err_mask; |
| 1510 | 1511 | bool dma = false; |
| 1511 | 1512 | |
| 1512 | 1513 | DPRINTK("read log page - log 0x%x, page 0x%x\n", log, page); |
| 1513 | 1514 | |
| 1515 | + /* | |
| 1516 | + * Return error without actually issuing the command on controllers | |
| 1517 | + * which e.g. lockup on a read log page. | |
| 1518 | + */ | |
| 1519 | + if (ap_flags & ATA_FLAG_NO_LOG_PAGE) | |
| 1520 | + return AC_ERR_DEV; | |
| 1521 | + | |
| 1514 | 1522 | retry: |
| 1515 | 1523 | ata_tf_init(dev, &tf); |
| 1516 | 1524 | if (dev->dma_mode && ata_id_has_read_log_dma_ext(dev->id) && |
| @@ -45,7 +45,8 @@ enum { · 44 unmodified lines | ||
| 45 | 45 | SATA_FSL_MAX_PRD_DIRECT = 16, /* Direct PRDT entries */ |
| 46 | 46 | |
| 47 | 47 | SATA_FSL_HOST_FLAGS = (ATA_FLAG_SATA | ATA_FLAG_PIO_DMA | |
| 48 | − ATA_FLAG_PMP | ATA_FLAG_NCQ | ATA_FLAG_AN), | |
| 48 | + ATA_FLAG_PMP | ATA_FLAG_NCQ | | |
| 49 | + ATA_FLAG_AN | ATA_FLAG_NO_LOG_PAGE), | |
| 49 | 50 | |
| 50 | 51 | SATA_FSL_MAX_CMDS = SATA_FSL_QUEUE_DEPTH, |
| 51 | 52 | SATA_FSL_CMD_HDR_SIZE = 16, /* 4 DWORDS */ |
| @@ -630,6 +630,9 @@ static void sil_dev_config(struct ata_device *dev) · 629 unmodified lines | ||
| 630 | 630 | unsigned int n, quirks = 0; |
| 631 | 631 | unsigned char model_num[ATA_ID_PROD_LEN + 1]; |
| 632 | 632 | |
| 633 | + /* This controller doesn't support trim */ | |
| 634 | + dev->horkage |= ATA_HORKAGE_NOTRIM; | |
| 635 | + | |
| 633 | 636 | ata_id_c_string(dev->id, model_num, ATA_ID_PROD, sizeof(model_num)); |
| 634 | 637 | |
| 635 | 638 | for (n = 0; sil_blacklist[n].product; n++) |
| @@ -303,6 +303,10 @@ static int memory_subsys_offline(struct device *dev) · 302 unmodified lines | ||
| 303 | 303 | if (mem->state == MEM_OFFLINE) |
| 304 | 304 | return 0; |
| 305 | 305 | |
| 306 | + /* Can't offline block with non-present sections */ | |
| 307 | + if (mem->section_count != sections_per_block) | |
| 308 | + return -EINVAL; | |
| 309 | + | |
| 306 | 310 | return memory_block_change_state(mem, MEM_OFFLINE, MEM_ONLINE); |
| 307 | 311 | } |
| 308 | 312 | |
| @@ -1775,10 +1775,10 @@ int genpd_dev_pm_attach(struct device *dev) · 1774 unmodified lines | ||
| 1775 | 1775 | } |
| 1776 | 1776 | |
| 1777 | 1777 | pd = of_genpd_get_from_provider(&pd_args); |
| 1778 | + of_node_put(pd_args.np); | |
| 1778 | 1779 | if (IS_ERR(pd)) { |
| 1779 | 1780 | dev_dbg(dev, "%s() failed to find PM domain: %ld\n", |
| 1780 | 1781 | __func__, PTR_ERR(pd)); |
| 1781 | − of_node_put(dev->of_node); | |
| 1782 | 1782 | return -EPROBE_DEFER; |
| 1783 | 1783 | } |
| 1784 | 1784 | |
| @@ -1796,7 +1796,6 @@ int genpd_dev_pm_attach(struct device *dev) · 11 unmodified lines | ||
| 1796 | 1796 | if (ret < 0) { |
| 1797 | 1797 | dev_err(dev, "failed to add to PM domain %s: %d", |
| 1798 | 1798 | pd->name, ret); |
| 1799 | − of_node_put(dev->of_node); | |
| 1800 | 1799 | goto out; |
| 1801 | 1800 | } |
| 1802 | 1801 | |
| @@ -160,9 +160,6 @@ static bool default_power_down_ok(struct dev_pm_domain *pd) · 159 unmodified lines | ||
| 160 | 160 | struct gpd_timing_data *td; |
| 161 | 161 | s64 constraint_ns; |
| 162 | 162 | |
| 163 | − if (!pdd->dev->driver) | |
| 164 | − continue; | |
| 165 | − | |
| 166 | 163 | /* |
| 167 | 164 | * Check if the device is allowed to be off long enough for the |
| 168 | 165 | * domain to turn off and on (that's how much time it will |
| @@ -3810,7 +3810,6 @@ static int mtip_block_initialize(struct driver_data *dd) · 3809 unmodified lines | ||
| 3810 | 3810 | sector_t capacity; |
| 3811 | 3811 | unsigned int index = 0; |
| 3812 | 3812 | struct kobject *kobj; |
| 3813 | − unsigned char thd_name[16]; | |
| 3814 | 3813 | |
| 3815 | 3814 | if (dd->disk) |
| 3816 | 3815 | goto skip_create_disk; /* hw init done, before rebuild */ |
| @@ -3958,10 +3957,9 @@ skip_create_disk: · 141 unmodified lines | ||
| 3958 | 3957 | } |
| 3959 | 3958 | |
| 3960 | 3959 | start_service_thread: |
| 3961 | − sprintf(thd_name, "mtip_svc_thd_%02d", index); | |
| 3962 | 3960 | dd->mtip_svc_handler = kthread_create_on_node(mtip_service_thread, |
| 3963 | − dd, dd->numa_node, "%s", | |
| 3964 | − thd_name); | |
| 3961 | + dd, dd->numa_node, | |
| 3962 | + "mtip_svc_thd_%02d", index); | |
| 3965 | 3963 | |
| 3966 | 3964 | if (IS_ERR(dd->mtip_svc_handler)) { |
| 3967 | 3965 | dev_err(&dd->pdev->dev, "service thread failed to start\n"); |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/slab.h> |
| 9 | 9 | #include <linux/blk-mq.h> |
| 10 | 10 | #include <linux/hrtimer.h> |
| 11 | +#include <linux/lightnvm.h> | |
| 11 | 12 | |
| 12 | 13 | struct nullb_cmd { |
| 13 | 14 | struct list_head list; |
| @@ -17,6 +18,7 @@ struct nullb_cmd { · 3 unmodified lines | ||
| 17 | 18 | struct bio *bio; |
| 18 | 19 | unsigned int tag; |
| 19 | 20 | struct nullb_queue *nq; |
| 21 | + struct hrtimer timer; | |
| 20 | 22 | }; |
| 21 | 23 | |
| 22 | 24 | struct nullb_queue { |
| @@ -39,23 +41,14 @@ struct nullb { · 16 unmodified lines | ||
| 39 | 41 | |
| 40 | 42 | struct nullb_queue *queues; |
| 41 | 43 | unsigned int nr_queues; |
| 44 | + char disk_name[DISK_NAME_LEN]; | |
| 42 | 45 | }; |
| 43 | 46 | |
| 44 | 47 | static LIST_HEAD(nullb_list); |
| 45 | 48 | static struct mutex lock; |
| 46 | 49 | static int null_major; |
| 47 | 50 | static int nullb_indexes; |
| 48 | − | |
| 49 | −struct completion_queue { | |
| 50 | − struct llist_head list; | |
| 51 | − struct hrtimer timer; | |
| 52 | −}; | |
| 53 | − | |
| 54 | −/* | |
| 55 | − * These are per-cpu for now, they will need to be configured by the | |
| 56 | − * complete_queues parameter and appropriately mapped. | |
| 57 | − */ | |
| 58 | −static DEFINE_PER_CPU(struct completion_queue, completion_queues); | |
| 51 | +static struct kmem_cache *ppa_cache; | |
| 59 | 52 | |
| 60 | 53 | enum { |
| 61 | 54 | NULL_IRQ_NONE = 0, |
| @@ -119,6 +112,10 @@ static int nr_devices = 2; · 57 unmodified lines | ||
| 119 | 112 | module_param(nr_devices, int, S_IRUGO); |
| 120 | 113 | MODULE_PARM_DESC(nr_devices, "Number of devices to register"); |
| 121 | 114 | |
| 115 | +static bool use_lightnvm; | |
| 116 | +module_param(use_lightnvm, bool, S_IRUGO); | |
| 117 | +MODULE_PARM_DESC(use_lightnvm, "Register as a LightNVM device"); | |
| 118 | + | |
| 122 | 119 | static int irqmode = NULL_IRQ_SOFTIRQ; |
| 123 | 120 | |
| 124 | 121 | static int null_set_irqmode(const char *str, const struct kernel_param *kp) |
| @@ -135,8 +132,8 @@ static const struct kernel_param_ops null_irqmode_param_ops = { · 10 unmodified lines | ||
| 135 | 132 | device_param_cb(irqmode, &null_irqmode_param_ops, &irqmode, S_IRUGO); |
| 136 | 133 | MODULE_PARM_DESC(irqmode, "IRQ completion handler. 0-none, 1-softirq, 2-timer"); |
| 137 | 134 | |
| 138 | −static int completion_nsec = 10000; | |
| 139 | −module_param(completion_nsec, int, S_IRUGO); | |
| 135 | +static unsigned long completion_nsec = 10000; | |
| 136 | +module_param(completion_nsec, ulong, S_IRUGO); | |
| 140 | 137 | MODULE_PARM_DESC(completion_nsec, "Time in ns to complete a request in hardware. Default: 10,000ns"); |
| 141 | 138 | |
| 142 | 139 | static int hw_queue_depth = 64; |
| @@ -173,6 +170,8 @@ static void free_cmd(struct nullb_cmd *cmd) · 30 unmodified lines | ||
| 173 | 170 | put_tag(cmd->nq, cmd->tag); |
| 174 | 171 | } |
| 175 | 172 | |
| 173 | +static enum hrtimer_restart null_cmd_timer_expired(struct hrtimer *timer); | |
| 174 | + | |
| 176 | 175 | static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq) |
| 177 | 176 | { |
| 178 | 177 | struct nullb_cmd *cmd; |
| @@ -183,6 +182,11 @@ static struct nullb_cmd *__alloc_cmd(struct nullb_queue *nq) · 4 unmodified lines | ||
| 183 | 182 | cmd = &nq->cmds[tag]; |
| 184 | 183 | cmd->tag = tag; |
| 185 | 184 | cmd->nq = nq; |
| 185 | + if (irqmode == NULL_IRQ_TIMER) { | |
| 186 | + hrtimer_init(&cmd->timer, CLOCK_MONOTONIC, | |
| 187 | + HRTIMER_MODE_REL); | |
| 188 | + cmd->timer.function = null_cmd_timer_expired; | |
| 189 | + } | |
| 186 | 190 | return cmd; |
| 187 | 191 | } |
| 188 | 192 | |
| @@ -213,6 +217,8 @@ static struct nullb_cmd *alloc_cmd(struct nullb_queue *nq, int can_wait) · 24 unmodified lines | ||
| 213 | 217 | |
| 214 | 218 | static void end_cmd(struct nullb_cmd *cmd) |
| 215 | 219 | { |
| 220 | + struct request_queue *q = NULL; | |
| 221 | + | |
| 216 | 222 | switch (queue_mode) { |
| 217 | 223 | case NULL_Q_MQ: |
| 218 | 224 | blk_mq_end_request(cmd->rq, 0); |
| @@ -223,55 +229,37 @@ static void end_cmd(struct nullb_cmd *cmd) · 4 unmodified lines | ||
| 223 | 229 | break; |
| 224 | 230 | case NULL_Q_BIO: |
| 225 | 231 | bio_endio(cmd->bio); |
| 226 | − break; | |
| 232 | + goto free_cmd; | |
| 227 | 233 | } |
| 228 | 234 | |
| 235 | + if (cmd->rq) | |
| 236 | + q = cmd->rq->q; | |
| 237 | + | |
| 238 | + /* Restart queue if needed, as we are freeing a tag */ | |
| 239 | + if (q && !q->mq_ops && blk_queue_stopped(q)) { | |
| 240 | + unsigned long flags; | |
| 241 | + | |
| 242 | + spin_lock_irqsave(q->queue_lock, flags); | |
| 243 | + if (blk_queue_stopped(q)) | |
| 244 | + blk_start_queue(q); | |
| 245 | + spin_unlock_irqrestore(q->queue_lock, flags); | |
| 246 | + } | |
| 247 | +free_cmd: | |
| 229 | 248 | free_cmd(cmd); |
| 230 | 249 | } |
| 231 | 250 | |
| 232 | 251 | static enum hrtimer_restart null_cmd_timer_expired(struct hrtimer *timer) |
| 233 | 252 | { |
| 234 | − struct completion_queue *cq; | |
| 235 | − struct llist_node *entry; | |
| 236 | − struct nullb_cmd *cmd; | |
| 237 | − | |
| 238 | − cq = &per_cpu(completion_queues, smp_processor_id()); | |
| 239 | − | |
| 240 | − while ((entry = llist_del_all(&cq->list)) != NULL) { | |
| 241 | − entry = llist_reverse_order(entry); | |
| 242 | − do { | |
| 243 | − struct request_queue *q = NULL; | |
| 244 | − | |
| 245 | − cmd = container_of(entry, struct nullb_cmd, ll_list); | |
| 246 | − entry = entry->next; | |
| 247 | − if (cmd->rq) | |
| 248 | − q = cmd->rq->q; | |
| 249 | − end_cmd(cmd); | |
| 250 | − | |
| 251 | − if (q && !q->mq_ops && blk_queue_stopped(q)) { | |
| 252 | − spin_lock(q->queue_lock); | |
| 253 | − if (blk_queue_stopped(q)) | |
| 254 | − blk_start_queue(q); | |
| 255 | − spin_unlock(q->queue_lock); | |
| 256 | − } | |
| 257 | − } while (entry); | |
| 258 | − } | |
| 253 | + end_cmd(container_of(timer, struct nullb_cmd, timer)); | |
| 259 | 254 | |
| 260 | 255 | return HRTIMER_NORESTART; |
| 261 | 256 | } |
| 262 | 257 | |
| 263 | 258 | static void null_cmd_end_timer(struct nullb_cmd *cmd) |
| 264 | 259 | { |
| 265 | − struct completion_queue *cq = &per_cpu(completion_queues, get_cpu()); | |
| 260 | + ktime_t kt = ktime_set(0, completion_nsec); | |
| 266 | 261 | |
| 267 | − cmd->ll_list.next = NULL; | |
| 268 | − if (llist_add(&cmd->ll_list, &cq->list)) { | |
| 269 | − ktime_t kt = ktime_set(0, completion_nsec); | |
| 270 | − | |
| 271 | − hrtimer_start(&cq->timer, kt, HRTIMER_MODE_REL_PINNED); | |
| 272 | − } | |
| 273 | − | |
| 274 | − put_cpu(); | |
| 262 | + hrtimer_start(&cmd->timer, kt, HRTIMER_MODE_REL); | |
| 275 | 263 | } |
| 276 | 264 | |
| 277 | 265 | static void null_softirq_done_fn(struct request *rq) |
| @@ -369,6 +357,10 @@ static int null_queue_rq(struct blk_mq_hw_ctx *hctx, · 91 unmodified lines | ||
| 369 | 357 | { |
| 370 | 358 | struct nullb_cmd *cmd = blk_mq_rq_to_pdu(bd->rq); |
| 371 | 359 | |
| 360 | + if (irqmode == NULL_IRQ_TIMER) { | |
| 361 | + hrtimer_init(&cmd->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); | |
| 362 | + cmd->timer.function = null_cmd_timer_expired; | |
| 363 | + } | |
| 372 | 364 | cmd->rq = bd->rq; |
| 373 | 365 | cmd->nq = hctx->driver_data; |
| 374 | 366 | |
| @@ -427,15 +419,157 @@ static void null_del_dev(struct nullb *nullb) · 52 unmodified lines | ||
| 427 | 419 | { |
| 428 | 420 | list_del_init(&nullb->list); |
| 429 | 421 | |
| 430 | − del_gendisk(nullb->disk); | |
| 422 | + if (use_lightnvm) | |
| 423 | + nvm_unregister(nullb->disk_name); | |
| 424 | + else | |
| 425 | + del_gendisk(nullb->disk); | |
| 431 | 426 | blk_cleanup_queue(nullb->q); |
| 432 | 427 | if (queue_mode == NULL_Q_MQ) |
| 433 | 428 | blk_mq_free_tag_set(&nullb->tag_set); |
| 434 | − put_disk(nullb->disk); | |
| 429 | + if (!use_lightnvm) | |
| 430 | + put_disk(nullb->disk); | |
| 435 | 431 | cleanup_queues(nullb); |
| 436 | 432 | kfree(nullb); |
| 437 | 433 | } |
| 438 | 434 | |
| 435 | +#ifdef CONFIG_NVM | |
| 436 | + | |
| 437 | +static void null_lnvm_end_io(struct request *rq, int error) | |
| 438 | +{ | |
| 439 | + struct nvm_rq *rqd = rq->end_io_data; | |
| 440 | + struct nvm_dev *dev = rqd->dev; | |
| 441 | + | |
| 442 | + dev->mt->end_io(rqd, error); | |
| 443 | + | |
| 444 | + blk_put_request(rq); | |
| 445 | +} | |
| 446 | + | |
| 447 | +static int null_lnvm_submit_io(struct nvm_dev *dev, struct nvm_rq *rqd) | |
| 448 | +{ | |
| 449 | + struct request_queue *q = dev->q; | |
| 450 | + struct request *rq; | |
| 451 | + struct bio *bio = rqd->bio; | |
| 452 | + | |
| 453 | + rq = blk_mq_alloc_request(q, bio_rw(bio), GFP_KERNEL, 0); | |
| 454 | + if (IS_ERR(rq)) | |
| 455 | + return -ENOMEM; | |
| 456 | + | |
| 457 | + rq->cmd_type = REQ_TYPE_DRV_PRIV; | |
| 458 | + rq->__sector = bio->bi_iter.bi_sector; | |
| 459 | + rq->ioprio = bio_prio(bio); | |
| 460 | + | |
| 461 | + if (bio_has_data(bio)) | |
| 462 | + rq->nr_phys_segments = bio_phys_segments(q, bio); | |
| 463 | + | |
| 464 | + rq->__data_len = bio->bi_iter.bi_size; | |
| 465 | + rq->bio = rq->biotail = bio; | |
| 466 | + | |
| 467 | + rq->end_io_data = rqd; | |
| 468 | + | |
| 469 | + blk_execute_rq_nowait(q, NULL, rq, 0, null_lnvm_end_io); | |
| 470 | + | |
| 471 | + return 0; | |
| 472 | +} | |
| 473 | + | |
| 474 | +static int null_lnvm_id(struct nvm_dev *dev, struct nvm_id *id) | |
| 475 | +{ | |
| 476 | + sector_t size = gb * 1024 * 1024 * 1024ULL; | |
| 477 | + sector_t blksize; | |
| 478 | + struct nvm_id_group *grp; | |
| 479 | + | |
| 480 | + id->ver_id = 0x1; | |
| 481 | + id->vmnt = 0; | |
| 482 | + id->cgrps = 1; | |
| 483 | + id->cap = 0x3; | |
| 484 | + id->dom = 0x1; | |
| 485 | + | |
| 486 | + id->ppaf.blk_offset = 0; | |
| 487 | + id->ppaf.blk_len = 16; | |
| 488 | + id->ppaf.pg_offset = 16; | |
| 489 | + id->ppaf.pg_len = 16; | |
| 490 | + id->ppaf.sect_offset = 32; | |
| 491 | + id->ppaf.sect_len = 8; | |
| 492 | + id->ppaf.pln_offset = 40; | |
| 493 | + id->ppaf.pln_len = 8; | |
| 494 | + id->ppaf.lun_offset = 48; | |
| 495 | + id->ppaf.lun_len = 8; | |
| 496 | + id->ppaf.ch_offset = 56; | |
| 497 | + id->ppaf.ch_len = 8; | |
| 498 | + | |
| 499 | + do_div(size, bs); /* convert size to pages */ | |
| 500 | + do_div(size, 256); /* concert size to pgs pr blk */ | |
| 501 | + grp = &id->groups[0]; | |
| 502 | + grp->mtype = 0; | |
| 503 | + grp->fmtype = 0; | |
| 504 | + grp->num_ch = 1; | |
| 505 | + grp->num_pg = 256; | |
| 506 | + blksize = size; | |
| 507 | + do_div(size, (1 << 16)); | |
| 508 | + grp->num_lun = size + 1; | |
| 509 | + do_div(blksize, grp->num_lun); | |
| 510 | + grp->num_blk = blksize; | |
| 511 | + grp->num_pln = 1; | |
| 512 | + | |
| 513 | + grp->fpg_sz = bs; | |
| 514 | + grp->csecs = bs; | |
| 515 | + grp->trdt = 25000; | |
| 516 | + grp->trdm = 25000; | |
| 517 | + grp->tprt = 500000; | |
| 518 | + grp->tprm = 500000; | |
| 519 | + grp->tbet = 1500000; | |
| 520 | + grp->tbem = 1500000; | |
| 521 | + grp->mpos = 0x010101; /* single plane rwe */ | |
| 522 | + grp->cpar = hw_queue_depth; | |
| 523 | + | |
| 524 | + return 0; | |
| 525 | +} | |
| 526 | + | |
| 527 | +static void *null_lnvm_create_dma_pool(struct nvm_dev *dev, char *name) | |
| 528 | +{ | |
| 529 | + mempool_t *virtmem_pool; | |
| 530 | + | |
| 531 | + virtmem_pool = mempool_create_slab_pool(64, ppa_cache); | |
| 532 | + if (!virtmem_pool) { | |
| 533 | + pr_err("null_blk: Unable to create virtual memory pool\n"); | |
| 534 | + return NULL; | |
| 535 | + } | |
| 536 | + | |
| 537 | + return virtmem_pool; | |
| 538 | +} | |
| 539 | + | |
| 540 | +static void null_lnvm_destroy_dma_pool(void *pool) | |
| 541 | +{ | |
| 542 | + mempool_destroy(pool); | |
| 543 | +} | |
| 544 | + | |
| 545 | +static void *null_lnvm_dev_dma_alloc(struct nvm_dev *dev, void *pool, | |
| 546 | + gfp_t mem_flags, dma_addr_t *dma_handler) | |
| 547 | +{ | |
| 548 | + return mempool_alloc(pool, mem_flags); | |
| 549 | +} | |
| 550 | + | |
| 551 | +static void null_lnvm_dev_dma_free(void *pool, void *entry, | |
| 552 | + dma_addr_t dma_handler) | |
| 553 | +{ | |
| 554 | + mempool_free(entry, pool); | |
| 555 | +} | |
| 556 | + | |
| 557 | +static struct nvm_dev_ops null_lnvm_dev_ops = { | |
| 558 | + .identity = null_lnvm_id, | |
| 559 | + .submit_io = null_lnvm_submit_io, | |
| 560 | + | |
| 561 | + .create_dma_pool = null_lnvm_create_dma_pool, | |
| 562 | + .destroy_dma_pool = null_lnvm_destroy_dma_pool, | |
| 563 | + .dev_dma_alloc = null_lnvm_dev_dma_alloc, | |
| 564 | + .dev_dma_free = null_lnvm_dev_dma_free, | |
| 565 | + | |
| 566 | + /* Simulate nvme protocol restriction */ | |
| 567 | + .max_phys_sect = 64, | |
| 568 | +}; | |
| 569 | +#else | |
| 570 | +static struct nvm_dev_ops null_lnvm_dev_ops; | |
| 571 | +#endif /* CONFIG_NVM */ | |
| 572 | + | |
| 439 | 573 | static int null_open(struct block_device *bdev, fmode_t mode) |
| 440 | 574 | { |
| 441 | 575 | return 0; |
| @@ -575,11 +709,6 @@ static int null_add_dev(void) · 133 unmodified lines | ||
| 575 | 709 | queue_flag_set_unlocked(QUEUE_FLAG_NONROT, nullb->q); |
| 576 | 710 | queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, nullb->q); |
| 577 | 711 | |
| 578 | − disk = nullb->disk = alloc_disk_node(1, home_node); | |
| 579 | − if (!disk) { | |
| 580 | − rv = -ENOMEM; | |
| 581 | − goto out_cleanup_blk_queue; | |
| 582 | − } | |
| 583 | 712 | |
| 584 | 713 | mutex_lock(&lock); |
| 585 | 714 | list_add_tail(&nullb->list, &nullb_list); |
| @@ -589,6 +718,21 @@ static int null_add_dev(void) · 3 unmodified lines | ||
| 589 | 718 | blk_queue_logical_block_size(nullb->q, bs); |
| 590 | 719 | blk_queue_physical_block_size(nullb->q, bs); |
| 591 | 720 | |
| 721 | + sprintf(nullb->disk_name, "nullb%d", nullb->index); | |
| 722 | + | |
| 723 | + if (use_lightnvm) { | |
| 724 | + rv = nvm_register(nullb->q, nullb->disk_name, | |
| 725 | + &null_lnvm_dev_ops); | |
| 726 | + if (rv) | |
| 727 | + goto out_cleanup_blk_queue; | |
| 728 | + goto done; | |
| 729 | + } | |
| 730 | + | |
| 731 | + disk = nullb->disk = alloc_disk_node(1, home_node); | |
| 732 | + if (!disk) { | |
| 733 | + rv = -ENOMEM; | |
| 734 | + goto out_cleanup_lightnvm; | |
| 735 | + } | |
| 592 | 736 | size = gb * 1024 * 1024 * 1024ULL; |
| 593 | 737 | set_capacity(disk, size >> 9); |
| 594 | 738 | |
| @@ -598,10 +742,15 @@ static int null_add_dev(void) · 3 unmodified lines | ||
| 598 | 742 | disk->fops = &null_fops; |
| 599 | 743 | disk->private_data = nullb; |
| 600 | 744 | disk->queue = nullb->q; |
| 601 | − sprintf(disk->disk_name, "nullb%d", nullb->index); | |
| 745 | + strncpy(disk->disk_name, nullb->disk_name, DISK_NAME_LEN); | |
| 746 | + | |
| 602 | 747 | add_disk(disk); |
| 748 | +done: | |
| 603 | 749 | return 0; |
| 604 | 750 | |
| 751 | +out_cleanup_lightnvm: | |
| 752 | + if (use_lightnvm) | |
| 753 | + nvm_unregister(nullb->disk_name); | |
| 605 | 754 | out_cleanup_blk_queue: |
| 606 | 755 | blk_cleanup_queue(nullb->q); |
| 607 | 756 | out_cleanup_tags: |
| @@ -617,7 +766,9 @@ out: · 9 unmodified lines | ||
| 617 | 766 | |
| 618 | 767 | static int __init null_init(void) |
| 619 | 768 | { |
| 769 | + int ret = 0; | |
| 620 | 770 | unsigned int i; |
| 771 | + struct nullb *nullb; | |
| 621 | 772 | |
| 622 | 773 | if (bs > PAGE_SIZE) { |
| 623 | 774 | pr_warn("null_blk: invalid block size\n"); |
| @@ -625,6 +776,18 @@ static int __init null_init(void) · 1 unmodified line | ||
| 625 | 776 | bs = PAGE_SIZE; |
| 626 | 777 | } |
| 627 | 778 | |
| 779 | + if (use_lightnvm && bs != 4096) { | |
| 780 | + pr_warn("null_blk: LightNVM only supports 4k block size\n"); | |
| 781 | + pr_warn("null_blk: defaults block size to 4k\n"); | |
| 782 | + bs = 4096; | |
| 783 | + } | |
| 784 | + | |
| 785 | + if (use_lightnvm && queue_mode != NULL_Q_MQ) { | |
| 786 | + pr_warn("null_blk: LightNVM only supported for blk-mq\n"); | |
| 787 | + pr_warn("null_blk: defaults queue mode to blk-mq\n"); | |
| 788 | + queue_mode = NULL_Q_MQ; | |
| 789 | + } | |
| 790 | + | |
| 628 | 791 | if (queue_mode == NULL_Q_MQ && use_per_node_hctx) { |
| 629 | 792 | if (submit_queues < nr_online_nodes) { |
| 630 | 793 | pr_warn("null_blk: submit_queues param is set to %u.", |
| @@ -638,32 +801,38 @@ static int __init null_init(void) · 7 unmodified lines | ||
| 638 | 801 | |
| 639 | 802 | mutex_init(&lock); |
| 640 | 803 | |
| 641 | − /* Initialize a separate list for each CPU for issuing softirqs */ | |
| 642 | − for_each_possible_cpu(i) { | |
| 643 | − struct completion_queue *cq = &per_cpu(completion_queues, i); | |
| 644 | − | |
| 645 | − init_llist_head(&cq->list); | |
| 646 | − | |
| 647 | − if (irqmode != NULL_IRQ_TIMER) | |
| 648 | − continue; | |
| 649 | − | |
| 650 | − hrtimer_init(&cq->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); | |
| 651 | − cq->timer.function = null_cmd_timer_expired; | |
| 652 | − } | |
| 653 | − | |
| 654 | 804 | null_major = register_blkdev(0, "nullb"); |
| 655 | 805 | if (null_major < 0) |
| 656 | 806 | return null_major; |
| 657 | 807 | |
| 658 | − for (i = 0; i < nr_devices; i++) { | |
| 659 | − if (null_add_dev()) { | |
| 660 | − unregister_blkdev(null_major, "nullb"); | |
| 661 | − return -EINVAL; | |
| 808 | + if (use_lightnvm) { | |
| 809 | + ppa_cache = kmem_cache_create("ppa_cache", 64 * sizeof(u64), | |
| 810 | + 0, 0, NULL); | |
| 811 | + if (!ppa_cache) { | |
| 812 | + pr_err("null_blk: unable to create ppa cache\n"); | |
| 813 | + ret = -ENOMEM; | |
| 814 | + goto err_ppa; | |
| 662 | 815 | } |
| 663 | 816 | } |
| 664 | 817 | |
| 818 | + for (i = 0; i < nr_devices; i++) { | |
| 819 | + ret = null_add_dev(); | |
| 820 | + if (ret) | |
| 821 | + goto err_dev; | |
| 822 | + } | |
| 823 | + | |
| 665 | 824 | pr_info("null: module loaded\n"); |
| 666 | 825 | return 0; |
| 826 | + | |
| 827 | +err_dev: | |
| 828 | + while (!list_empty(&nullb_list)) { | |
| 829 | + nullb = list_entry(nullb_list.next, struct nullb, list); | |
| 830 | + null_del_dev(nullb); | |
| 831 | + } | |
| 832 | + kmem_cache_destroy(ppa_cache); | |
| 833 | +err_ppa: | |
| 834 | + unregister_blkdev(null_major, "nullb"); | |
| 835 | + return ret; | |
| 667 | 836 | } |
| 668 | 837 | |
| 669 | 838 | static void __exit null_exit(void) |
| @@ -678,6 +847,8 @@ static void __exit null_exit(void) · 8 unmodified lines | ||
| 678 | 847 | null_del_dev(nullb); |
| 679 | 848 | } |
| 680 | 849 | mutex_unlock(&lock); |
| 850 | + | |
| 851 | + kmem_cache_destroy(ppa_cache); | |
| 681 | 852 | } |
| 682 | 853 | |
| 683 | 854 | module_init(null_init); |
| @@ -3442,6 +3442,7 @@ static void rbd_queue_workfn(struct work_struct *work) · 3441 unmodified lines | ||
| 3442 | 3442 | goto err_rq; |
| 3443 | 3443 | } |
| 3444 | 3444 | img_request->rq = rq; |
| 3445 | + snapc = NULL; /* img_request consumes a ref */ | |
| 3445 | 3446 | |
| 3446 | 3447 | if (op_type == OBJ_OP_DISCARD) |
| 3447 | 3448 | result = rbd_img_request_fill(img_request, OBJ_REQUEST_NODATA, |
| @@ -117,7 +117,7 @@ static struct platform_driver omap_ocp2scp_driver = { · 116 unmodified lines | ||
| 117 | 117 | |
| 118 | 118 | module_platform_driver(omap_ocp2scp_driver); |
| 119 | 119 | |
| 120 | −MODULE_ALIAS("platform: omap-ocp2scp"); | |
| 120 | +MODULE_ALIAS("platform:omap-ocp2scp"); | |
| 121 | 121 | MODULE_AUTHOR("Texas Instruments Inc."); |
| 122 | 122 | MODULE_DESCRIPTION("OMAP OCP2SCP driver"); |
| 123 | 123 | MODULE_LICENSE("GPL v2"); |
| @@ -1230,14 +1230,14 @@ static int smi_start_processing(void *send_info, · 1229 unmodified lines | ||
| 1230 | 1230 | |
| 1231 | 1231 | new_smi->intf = intf; |
| 1232 | 1232 | |
| 1233 | − /* Try to claim any interrupts. */ | |
| 1234 | − if (new_smi->irq_setup) | |
| 1235 | − new_smi->irq_setup(new_smi); | |
| 1236 | − | |
| 1237 | 1233 | /* Set up the timer that drives the interface. */ |
| 1238 | 1234 | setup_timer(&new_smi->si_timer, smi_timeout, (long)new_smi); |
| 1239 | 1235 | smi_mod_timer(new_smi, jiffies + SI_TIMEOUT_JIFFIES); |
| 1240 | 1236 | |
| 1237 | + /* Try to claim any interrupts. */ | |
| 1238 | + if (new_smi->irq_setup) | |
| 1239 | + new_smi->irq_setup(new_smi); | |
| 1240 | + | |
| 1241 | 1241 | /* |
| 1242 | 1242 | * Check if the user forcefully enabled the daemon. |
| 1243 | 1243 | */ |
| @@ -209,6 +209,8 @@ EXPORT_SYMBOL_GPL(clk_register_gpio_mux); · 208 unmodified lines | ||
| 209 | 209 | |
| 210 | 210 | struct clk_gpio_delayed_register_data { |
| 211 | 211 | const char *gpio_name; |
| 212 | + int num_parents; | |
| 213 | + const char **parent_names; | |
| 212 | 214 | struct device_node *node; |
| 213 | 215 | struct mutex lock; |
| 214 | 216 | struct clk *clk; |
| @@ -222,8 +224,6 @@ static struct clk *of_clk_gpio_delayed_register_get( · 7 unmodified lines | ||
| 222 | 224 | { |
| 223 | 225 | struct clk_gpio_delayed_register_data *data = _data; |
| 224 | 226 | struct clk *clk; |
| 225 | − const char **parent_names; | |
| 226 | − int i, num_parents; | |
| 227 | 227 | int gpio; |
| 228 | 228 | enum of_gpio_flags of_flags; |
| 229 | 229 | |
| @@ -248,26 +248,14 @@ static struct clk *of_clk_gpio_delayed_register_get( · 18 unmodified lines | ||
| 248 | 248 | return ERR_PTR(gpio); |
| 249 | 249 | } |
| 250 | 250 | |
| 251 | − num_parents = of_clk_get_parent_count(data->node); | |
| 252 | − | |
| 253 | − parent_names = kcalloc(num_parents, sizeof(char *), GFP_KERNEL); | |
| 254 | − if (!parent_names) { | |
| 255 | − clk = ERR_PTR(-ENOMEM); | |
| 256 | − goto out; | |
| 257 | − } | |
| 258 | − | |
| 259 | − for (i = 0; i < num_parents; i++) | |
| 260 | − parent_names[i] = of_clk_get_parent_name(data->node, i); | |
| 261 | − | |
| 262 | − clk = data->clk_register_get(data->node->name, parent_names, | |
| 263 | − num_parents, gpio, of_flags & OF_GPIO_ACTIVE_LOW); | |
| 251 | + clk = data->clk_register_get(data->node->name, data->parent_names, | |
| 252 | + data->num_parents, gpio, of_flags & OF_GPIO_ACTIVE_LOW); | |
| 264 | 253 | if (IS_ERR(clk)) |
| 265 | 254 | goto out; |
| 266 | 255 | |
| 267 | 256 | data->clk = clk; |
| 268 | 257 | out: |
| 269 | 258 | mutex_unlock(&data->lock); |
| 270 | − kfree(parent_names); | |
| 271 | 259 | |
| 272 | 260 | return clk; |
| 273 | 261 | } |
| @@ -296,11 +284,24 @@ static void __init of_gpio_clk_setup(struct device_node *node, · 22 unmodified lines | ||
| 296 | 284 | unsigned gpio, bool active_low)) |
| 297 | 285 | { |
| 298 | 286 | struct clk_gpio_delayed_register_data *data; |
| 287 | + const char **parent_names; | |
| 288 | + int i, num_parents; | |
| 299 | 289 | |
| 300 | 290 | data = kzalloc(sizeof(*data), GFP_KERNEL); |
| 301 | 291 | if (!data) |
| 302 | 292 | return; |
| 303 | 293 | |
| 294 | + num_parents = of_clk_get_parent_count(node); | |
| 295 | + | |
| 296 | + parent_names = kcalloc(num_parents, sizeof(char *), GFP_KERNEL); | |
| 297 | + if (!parent_names) | |
| 298 | + return; | |
| 299 | + | |
| 300 | + for (i = 0; i < num_parents; i++) | |
| 301 | + parent_names[i] = of_clk_get_parent_name(node, i); | |
| 302 | + | |
| 303 | + data->num_parents = num_parents; | |
| 304 | + data->parent_names = parent_names; | |
| 304 | 305 | data->node = node; |
| 305 | 306 | data->gpio_name = gpio_name; |
| 306 | 307 | data->clk_register_get = clk_register_get; |
| @@ -778,8 +778,10 @@ static struct clk * __init create_one_cmux(struct clockgen *cg, int idx) · 777 unmodified lines | ||
| 778 | 778 | */ |
| 779 | 779 | clksel = (cg_in(cg, hwc->reg) & CLKSEL_MASK) >> CLKSEL_SHIFT; |
| 780 | 780 | div = get_pll_div(cg, hwc, clksel); |
| 781 | − if (!div) | |
| 781 | + if (!div) { | |
| 782 | + kfree(hwc); | |
| 782 | 783 | return NULL; |
| 784 | + } | |
| 783 | 785 | |
| 784 | 786 | pct80_rate = clk_get_rate(div->clk); |
| 785 | 787 | pct80_rate *= 8; |
| @@ -292,6 +292,7 @@ static int scpi_clocks_probe(struct platform_device *pdev) · 291 unmodified lines | ||
| 292 | 292 | ret = scpi_clk_add(dev, child, match); |
| 293 | 293 | if (ret) { |
| 294 | 294 | scpi_clocks_remove(pdev); |
| 295 | + of_node_put(child); | |
| 295 | 296 | return ret; |
| 296 | 297 | } |
| 297 | 298 | } |
| @@ -52,7 +52,7 @@ static unsigned long clk_pllv1_recalc_rate(struct clk_hw *hw, · 51 unmodified lines | ||
| 52 | 52 | unsigned long parent_rate) |
| 53 | 53 | { |
| 54 | 54 | struct clk_pllv1 *pll = to_clk_pllv1(hw); |
| 55 | − long long ll; | |
| 55 | + unsigned long long ull; | |
| 56 | 56 | int mfn_abs; |
| 57 | 57 | unsigned int mfi, mfn, mfd, pd; |
| 58 | 58 | u32 reg; |
| @@ -94,16 +94,16 @@ static unsigned long clk_pllv1_recalc_rate(struct clk_hw *hw, · 35 unmodified lines | ||
| 94 | 94 | rate = parent_rate * 2; |
| 95 | 95 | rate /= pd + 1; |
| 96 | 96 | |
| 97 | − ll = (unsigned long long)rate * mfn_abs; | |
| 97 | + ull = (unsigned long long)rate * mfn_abs; | |
| 98 | 98 | |
| 99 | − do_div(ll, mfd + 1); | |
| 99 | + do_div(ull, mfd + 1); | |
| 100 | 100 | |
| 101 | 101 | if (mfn_is_negative(pll, mfn)) |
| 102 | − ll = -ll; | |
| 102 | + ull = (rate * mfi) - ull; | |
| 103 | + else | |
| 104 | + ull = (rate * mfi) + ull; | |
| 103 | 105 | |
| 104 | − ll = (rate * mfi) + ll; | |
| 105 | − | |
| 106 | − return ll; | |
| 106 | + return ull; | |
| 107 | 107 | } |
| 108 | 108 | |
| 109 | 109 | static struct clk_ops clk_pllv1_ops = { |
| @@ -79,7 +79,7 @@ static unsigned long __clk_pllv2_recalc_rate(unsigned long parent_rate, · 78 unmodified lines | ||
| 79 | 79 | { |
| 80 | 80 | long mfi, mfn, mfd, pdf, ref_clk; |
| 81 | 81 | unsigned long dbl; |
| 82 | − s64 temp; | |
| 82 | + u64 temp; | |
| 83 | 83 | |
| 84 | 84 | dbl = dp_ctl & MXC_PLL_DP_CTL_DPDCK0_2_EN; |
| 85 | 85 | |
| @@ -98,8 +98,9 @@ static unsigned long __clk_pllv2_recalc_rate(unsigned long parent_rate, · 12 unmodified lines | ||
| 98 | 98 | temp = (u64) ref_clk * abs(mfn); |
| 99 | 99 | do_div(temp, mfd + 1); |
| 100 | 100 | if (mfn < 0) |
| 101 | − temp = -temp; | |
| 102 | − temp = (ref_clk * mfi) + temp; | |
| 101 | + temp = (ref_clk * mfi) - temp; | |
| 102 | + else | |
| 103 | + temp = (ref_clk * mfi) + temp; | |
| 103 | 104 | |
| 104 | 105 | return temp; |
| 105 | 106 | } |
| @@ -126,7 +127,7 @@ static int __clk_pllv2_set_rate(unsigned long rate, unsigned long parent_rate, · 20 unmodified lines | ||
| 126 | 127 | { |
| 127 | 128 | u32 reg; |
| 128 | 129 | long mfi, pdf, mfn, mfd = 999999; |
| 129 | − s64 temp64; | |
| 130 | + u64 temp64; | |
| 130 | 131 | unsigned long quad_parent_rate; |
| 131 | 132 | |
| 132 | 133 | quad_parent_rate = 4 * parent_rate; |
| @@ -335,22 +335,22 @@ static void __init vf610_clocks_init(struct device_node *ccm_node) · 334 unmodified lines | ||
| 335 | 335 | clk[VF610_CLK_SAI0_SEL] = imx_clk_mux("sai0_sel", CCM_CSCMR1, 0, 2, sai_sels, 4); |
| 336 | 336 | clk[VF610_CLK_SAI0_EN] = imx_clk_gate("sai0_en", "sai0_sel", CCM_CSCDR1, 16); |
| 337 | 337 | clk[VF610_CLK_SAI0_DIV] = imx_clk_divider("sai0_div", "sai0_en", CCM_CSCDR1, 0, 4); |
| 338 | − clk[VF610_CLK_SAI0] = imx_clk_gate2("sai0", "sai0_div", CCM_CCGR0, CCM_CCGRx_CGn(15)); | |
| 338 | + clk[VF610_CLK_SAI0] = imx_clk_gate2("sai0", "ipg_bus", CCM_CCGR0, CCM_CCGRx_CGn(15)); | |
| 339 | 339 | |
| 340 | 340 | clk[VF610_CLK_SAI1_SEL] = imx_clk_mux("sai1_sel", CCM_CSCMR1, 2, 2, sai_sels, 4); |
| 341 | 341 | clk[VF610_CLK_SAI1_EN] = imx_clk_gate("sai1_en", "sai1_sel", CCM_CSCDR1, 17); |
| 342 | 342 | clk[VF610_CLK_SAI1_DIV] = imx_clk_divider("sai1_div", "sai1_en", CCM_CSCDR1, 4, 4); |
| 343 | − clk[VF610_CLK_SAI1] = imx_clk_gate2("sai1", "sai1_div", CCM_CCGR1, CCM_CCGRx_CGn(0)); | |
| 343 | + clk[VF610_CLK_SAI1] = imx_clk_gate2("sai1", "ipg_bus", CCM_CCGR1, CCM_CCGRx_CGn(0)); | |
| 344 | 344 | |
| 345 | 345 | clk[VF610_CLK_SAI2_SEL] = imx_clk_mux("sai2_sel", CCM_CSCMR1, 4, 2, sai_sels, 4); |
| 346 | 346 | clk[VF610_CLK_SAI2_EN] = imx_clk_gate("sai2_en", "sai2_sel", CCM_CSCDR1, 18); |
| 347 | 347 | clk[VF610_CLK_SAI2_DIV] = imx_clk_divider("sai2_div", "sai2_en", CCM_CSCDR1, 8, 4); |
| 348 | − clk[VF610_CLK_SAI2] = imx_clk_gate2("sai2", "sai2_div", CCM_CCGR1, CCM_CCGRx_CGn(1)); | |
| 348 | + clk[VF610_CLK_SAI2] = imx_clk_gate2("sai2", "ipg_bus", CCM_CCGR1, CCM_CCGRx_CGn(1)); | |
| 349 | 349 | |
| 350 | 350 | clk[VF610_CLK_SAI3_SEL] = imx_clk_mux("sai3_sel", CCM_CSCMR1, 6, 2, sai_sels, 4); |
| 351 | 351 | clk[VF610_CLK_SAI3_EN] = imx_clk_gate("sai3_en", "sai3_sel", CCM_CSCDR1, 19); |
| 352 | 352 | clk[VF610_CLK_SAI3_DIV] = imx_clk_divider("sai3_div", "sai3_en", CCM_CSCDR1, 12, 4); |
| 353 | − clk[VF610_CLK_SAI3] = imx_clk_gate2("sai3", "sai3_div", CCM_CCGR1, CCM_CCGRx_CGn(2)); | |
| 353 | + clk[VF610_CLK_SAI3] = imx_clk_gate2("sai3", "ipg_bus", CCM_CCGR1, CCM_CCGRx_CGn(2)); | |
| 354 | 354 | |
| 355 | 355 | clk[VF610_CLK_NFC_SEL] = imx_clk_mux("nfc_sel", CCM_CSCMR1, 12, 2, nfc_sels, 4); |
| 356 | 356 | clk[VF610_CLK_NFC_EN] = imx_clk_gate("nfc_en", "nfc_sel", CCM_CSCDR2, 9); |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | * warranty of any kind, whether express or implied. |
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | +#include <linux/clk.h> | |
| 12 | 13 | #include <linux/module.h> |
| 13 | 14 | #include <linux/kernel.h> |
| 14 | 15 | #include <linux/spinlock.h> |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | * warranty of any kind, whether express or implied. |
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | +#include <linux/clk.h> | |
| 12 | 13 | #include <linux/module.h> |
| 13 | 14 | #include <linux/kernel.h> |
| 14 | 15 | #include <linux/spinlock.h> |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | * warranty of any kind, whether express or implied. |
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | +#include <linux/clk.h> | |
| 12 | 13 | #include <linux/module.h> |
| 13 | 14 | #include <linux/kernel.h> |
| 14 | 15 | #include <linux/spinlock.h> |
| @@ -41,15 +41,10 @@ · 40 unmodified lines | ||
| 41 | 41 | |
| 42 | 42 | #define SUN4I_PLL2_OUTPUTS 4 |
| 43 | 43 | |
| 44 | −struct sun4i_pll2_data { | |
| 45 | − u32 post_div_offset; | |
| 46 | − u32 pre_div_flags; | |
| 47 | −}; | |
| 48 | − | |
| 49 | 44 | static DEFINE_SPINLOCK(sun4i_a10_pll2_lock); |
| 50 | 45 | |
| 51 | 46 | static void __init sun4i_pll2_setup(struct device_node *node, |
| 52 | − struct sun4i_pll2_data *data) | |
| 47 | + int post_div_offset) | |
| 53 | 48 | { |
| 54 | 49 | const char *clk_name = node->name, *parent; |
| 55 | 50 | struct clk **clks, *base_clk, *prediv_clk; |
| @@ -76,7 +71,7 @@ static void __init sun4i_pll2_setup(struct device_node *node, · 20 unmodified lines | ||
| 76 | 71 | parent, 0, reg, |
| 77 | 72 | SUN4I_PLL2_PRE_DIV_SHIFT, |
| 78 | 73 | SUN4I_PLL2_PRE_DIV_WIDTH, |
| 79 | − data->pre_div_flags, | |
| 74 | + CLK_DIVIDER_ONE_BASED | CLK_DIVIDER_ALLOW_ZERO, | |
| 80 | 75 | &sun4i_a10_pll2_lock); |
| 81 | 76 | if (!prediv_clk) { |
| 82 | 77 | pr_err("Couldn't register the prediv clock\n"); |
| @@ -127,7 +122,7 @@ static void __init sun4i_pll2_setup(struct device_node *node, · 44 unmodified lines | ||
| 127 | 122 | */ |
| 128 | 123 | val = readl(reg); |
| 129 | 124 | val &= ~(SUN4I_PLL2_POST_DIV_MASK << SUN4I_PLL2_POST_DIV_SHIFT); |
| 130 | − val |= (SUN4I_PLL2_POST_DIV_VALUE - data->post_div_offset) << SUN4I_PLL2_POST_DIV_SHIFT; | |
| 125 | + val |= (SUN4I_PLL2_POST_DIV_VALUE - post_div_offset) << SUN4I_PLL2_POST_DIV_SHIFT; | |
| 131 | 126 | writel(val, reg); |
| 132 | 127 | |
| 133 | 128 | of_property_read_string_index(node, "clock-output-names", |
| @@ -191,25 +186,17 @@ err_unmap: · 57 unmodified lines | ||
| 191 | 186 | iounmap(reg); |
| 192 | 187 | } |
| 193 | 188 | |
| 194 | −static struct sun4i_pll2_data sun4i_a10_pll2_data = { | |
| 195 | − .pre_div_flags = CLK_DIVIDER_ONE_BASED | CLK_DIVIDER_ALLOW_ZERO, | |
| 196 | −}; | |
| 197 | − | |
| 198 | 189 | static void __init sun4i_a10_pll2_setup(struct device_node *node) |
| 199 | 190 | { |
| 200 | − sun4i_pll2_setup(node, &sun4i_a10_pll2_data); | |
| 191 | + sun4i_pll2_setup(node, 0); | |
| 201 | 192 | } |
| 202 | 193 | |
| 203 | 194 | CLK_OF_DECLARE(sun4i_a10_pll2, "allwinner,sun4i-a10-pll2-clk", |
| 204 | 195 | sun4i_a10_pll2_setup); |
| 205 | 196 | |
| 206 | −static struct sun4i_pll2_data sun5i_a13_pll2_data = { | |
| 207 | − .post_div_offset = 1, | |
| 208 | −}; | |
| 209 | − | |
| 210 | 197 | static void __init sun5i_a13_pll2_setup(struct device_node *node) |
| 211 | 198 | { |
| 212 | − sun4i_pll2_setup(node, &sun5i_a13_pll2_data); | |
| 199 | + sun4i_pll2_setup(node, 1); | |
| 213 | 200 | } |
| 214 | 201 | |
| 215 | 202 | CLK_OF_DECLARE(sun5i_a13_pll2, "allwinner,sun5i-a13-pll2-clk", |
| @@ -20,6 +20,8 @@ static struct ti_dt_clk dm816x_clks[] = { · 19 unmodified lines | ||
| 20 | 20 | DT_CLK(NULL, "sys_clkin", "sys_clkin_ck"), |
| 21 | 21 | DT_CLK(NULL, "timer_sys_ck", "sys_clkin_ck"), |
| 22 | 22 | DT_CLK(NULL, "sys_32k_ck", "sys_32k_ck"), |
| 23 | + DT_CLK(NULL, "timer_32k_ck", "sysclk18_ck"), | |
| 24 | + DT_CLK(NULL, "timer_ext_ck", "tclkin_ck"), | |
| 23 | 25 | DT_CLK(NULL, "mpu_ck", "mpu_ck"), |
| 24 | 26 | DT_CLK(NULL, "timer1_fck", "timer1_fck"), |
| 25 | 27 | DT_CLK(NULL, "timer2_fck", "timer2_fck"), |
| @@ -240,7 +240,7 @@ u8 omap2_init_dpll_parent(struct clk_hw *hw) · 239 unmodified lines | ||
| 240 | 240 | */ |
| 241 | 241 | unsigned long omap2_get_dpll_rate(struct clk_hw_omap *clk) |
| 242 | 242 | { |
| 243 | − long long dpll_clk; | |
| 243 | + u64 dpll_clk; | |
| 244 | 244 | u32 dpll_mult, dpll_div, v; |
| 245 | 245 | struct dpll_data *dd; |
| 246 | 246 | |
| @@ -262,7 +262,7 @@ unsigned long omap2_get_dpll_rate(struct clk_hw_omap *clk) · 15 unmodified lines | ||
| 262 | 262 | dpll_div = v & dd->div1_mask; |
| 263 | 263 | dpll_div >>= __ffs(dd->div1_mask); |
| 264 | 264 | |
| 265 | − dpll_clk = (long long)clk_get_rate(dd->clk_ref) * dpll_mult; | |
| 265 | + dpll_clk = (u64)clk_get_rate(dd->clk_ref) * dpll_mult; | |
| 266 | 266 | do_div(dpll_clk, dpll_div + 1); |
| 267 | 267 | |
| 268 | 268 | return dpll_clk; |
| @@ -214,7 +214,6 @@ static int ti_clk_divider_set_rate(struct clk_hw *hw, unsigned long rate, · 213 unmodified lines | ||
| 214 | 214 | { |
| 215 | 215 | struct clk_divider *divider; |
| 216 | 216 | unsigned int div, value; |
| 217 | − unsigned long flags = 0; | |
| 218 | 217 | u32 val; |
| 219 | 218 | |
| 220 | 219 | if (!hw || !rate) |
| @@ -228,9 +227,6 @@ static int ti_clk_divider_set_rate(struct clk_hw *hw, unsigned long rate, · 7 unmodified lines | ||
| 228 | 227 | if (value > div_mask(divider)) |
| 229 | 228 | value = div_mask(divider); |
| 230 | 229 | |
| 231 | − if (divider->lock) | |
| 232 | − spin_lock_irqsave(divider->lock, flags); | |
| 233 | − | |
| 234 | 230 | if (divider->flags & CLK_DIVIDER_HIWORD_MASK) { |
| 235 | 231 | val = div_mask(divider) << (divider->shift + 16); |
| 236 | 232 | } else { |
| @@ -240,9 +236,6 @@ static int ti_clk_divider_set_rate(struct clk_hw *hw, unsigned long rate, · 3 unmodified lines | ||
| 240 | 236 | val |= value << divider->shift; |
| 241 | 237 | ti_clk_ll_ops->clk_writel(val, divider->reg); |
| 242 | 238 | |
| 243 | − if (divider->lock) | |
| 244 | − spin_unlock_irqrestore(divider->lock, flags); | |
| 245 | − | |
| 246 | 239 | return 0; |
| 247 | 240 | } |
| 248 | 241 | |
| @@ -256,8 +249,7 @@ static struct clk *_register_divider(struct device *dev, const char *name, · 7 unmodified lines | ||
| 256 | 249 | const char *parent_name, |
| 257 | 250 | unsigned long flags, void __iomem *reg, |
| 258 | 251 | u8 shift, u8 width, u8 clk_divider_flags, |
| 259 | − const struct clk_div_table *table, | |
| 260 | − spinlock_t *lock) | |
| 252 | + const struct clk_div_table *table) | |
| 261 | 253 | { |
| 262 | 254 | struct clk_divider *div; |
| 263 | 255 | struct clk *clk; |
| @@ -288,7 +280,6 @@ static struct clk *_register_divider(struct device *dev, const char *name, · 24 unmodified lines | ||
| 288 | 280 | div->shift = shift; |
| 289 | 281 | div->width = width; |
| 290 | 282 | div->flags = clk_divider_flags; |
| 291 | − div->lock = lock; | |
| 292 | 283 | div->hw.init = &init; |
| 293 | 284 | div->table = table; |
| 294 | 285 | |
| @@ -421,7 +412,7 @@ struct clk *ti_clk_register_divider(struct ti_clk *setup) · 126 unmodified lines | ||
| 421 | 412 | |
| 422 | 413 | clk = _register_divider(NULL, setup->name, div->parent, |
| 423 | 414 | flags, (void __iomem *)reg, div->bit_shift, |
| 424 | − width, div_flags, table, NULL); | |
| 415 | + width, div_flags, table); | |
| 425 | 416 | |
| 426 | 417 | if (IS_ERR(clk)) |
| 427 | 418 | kfree(table); |
| @@ -584,8 +575,7 @@ static void __init of_ti_divider_clk_setup(struct device_node *node) · 156 unmodified lines | ||
| 584 | 575 | goto cleanup; |
| 585 | 576 | |
| 586 | 577 | clk = _register_divider(NULL, node->name, parent_name, flags, reg, |
| 587 | − shift, width, clk_divider_flags, table, | |
| 588 | − NULL); | |
| 578 | + shift, width, clk_divider_flags, table); | |
| 589 | 579 | |
| 590 | 580 | if (!IS_ERR(clk)) { |
| 591 | 581 | of_clk_add_provider(node, of_clk_src_simple_get, clk); |
| @@ -168,7 +168,7 @@ static unsigned long ti_fapll_recalc_rate(struct clk_hw *hw, · 167 unmodified lines | ||
| 168 | 168 | { |
| 169 | 169 | struct fapll_data *fd = to_fapll(hw); |
| 170 | 170 | u32 fapll_n, fapll_p, v; |
| 171 | − long long rate; | |
| 171 | + u64 rate; | |
| 172 | 172 | |
| 173 | 173 | if (ti_fapll_clock_is_bypass(fd)) |
| 174 | 174 | return parent_rate; |
| @@ -314,7 +314,7 @@ static unsigned long ti_fapll_synth_recalc_rate(struct clk_hw *hw, · 139 unmodified lines | ||
| 314 | 314 | { |
| 315 | 315 | struct fapll_synth *synth = to_synth(hw); |
| 316 | 316 | u32 synth_div_m; |
| 317 | − long long rate; | |
| 317 | + u64 rate; | |
| 318 | 318 | |
| 319 | 319 | /* The audio_pll_clk1 is hardwired to produce 32.768KiHz clock */ |
| 320 | 320 | if (!synth->div) |
| @@ -69,7 +69,6 @@ static int ti_clk_mux_set_parent(struct clk_hw *hw, u8 index) · 68 unmodified lines | ||
| 69 | 69 | { |
| 70 | 70 | struct clk_mux *mux = to_clk_mux(hw); |
| 71 | 71 | u32 val; |
| 72 | − unsigned long flags = 0; | |
| 73 | 72 | |
| 74 | 73 | if (mux->table) { |
| 75 | 74 | index = mux->table[index]; |
| @@ -81,9 +80,6 @@ static int ti_clk_mux_set_parent(struct clk_hw *hw, u8 index) · 5 unmodified lines | ||
| 81 | 80 | index++; |
| 82 | 81 | } |
| 83 | 82 | |
| 84 | − if (mux->lock) | |
| 85 | − spin_lock_irqsave(mux->lock, flags); | |
| 86 | − | |
| 87 | 83 | if (mux->flags & CLK_MUX_HIWORD_MASK) { |
| 88 | 84 | val = mux->mask << (mux->shift + 16); |
| 89 | 85 | } else { |
| @@ -93,9 +89,6 @@ static int ti_clk_mux_set_parent(struct clk_hw *hw, u8 index) · 3 unmodified lines | ||
| 93 | 89 | val |= index << mux->shift; |
| 94 | 90 | ti_clk_ll_ops->clk_writel(val, mux->reg); |
| 95 | 91 | |
| 96 | − if (mux->lock) | |
| 97 | − spin_unlock_irqrestore(mux->lock, flags); | |
| 98 | − | |
| 99 | 92 | return 0; |
| 100 | 93 | } |
| 101 | 94 | |
| @@ -109,7 +102,7 @@ static struct clk *_register_mux(struct device *dev, const char *name, · 7 unmodified lines | ||
| 109 | 102 | const char **parent_names, u8 num_parents, |
| 110 | 103 | unsigned long flags, void __iomem *reg, |
| 111 | 104 | u8 shift, u32 mask, u8 clk_mux_flags, |
| 112 | − u32 *table, spinlock_t *lock) | |
| 105 | + u32 *table) | |
| 113 | 106 | { |
| 114 | 107 | struct clk_mux *mux; |
| 115 | 108 | struct clk *clk; |
| @@ -133,7 +126,6 @@ static struct clk *_register_mux(struct device *dev, const char *name, · 17 unmodified lines | ||
| 133 | 126 | mux->shift = shift; |
| 134 | 127 | mux->mask = mask; |
| 135 | 128 | mux->flags = clk_mux_flags; |
| 136 | − mux->lock = lock; | |
| 137 | 129 | mux->table = table; |
| 138 | 130 | mux->hw.init = &init; |
| 139 | 131 | |
| @@ -175,7 +167,7 @@ struct clk *ti_clk_register_mux(struct ti_clk *setup) · 35 unmodified lines | ||
| 175 | 167 | |
| 176 | 168 | return _register_mux(NULL, setup->name, mux->parents, mux->num_parents, |
| 177 | 169 | flags, (void __iomem *)reg, mux->bit_shift, mask, |
| 178 | − mux_flags, NULL, NULL); | |
| 170 | + mux_flags, NULL); | |
| 179 | 171 | } |
| 180 | 172 | |
| 181 | 173 | /** |
| @@ -227,8 +219,7 @@ static void of_mux_clk_setup(struct device_node *node) · 45 unmodified lines | ||
| 227 | 219 | mask = (1 << fls(mask)) - 1; |
| 228 | 220 | |
| 229 | 221 | clk = _register_mux(NULL, node->name, parent_names, num_parents, |
| 230 | − flags, reg, shift, mask, clk_mux_flags, NULL, | |
| 231 | − NULL); | |
| 222 | + flags, reg, shift, mask, clk_mux_flags, NULL); | |
| 232 | 223 | |
| 233 | 224 | if (!IS_ERR(clk)) |
| 234 | 225 | of_clk_add_provider(node, of_clk_src_simple_get, clk); |
| @@ -55,7 +55,7 @@ int __init clocksource_mmio_init(void __iomem *base, const char *name, · 54 unmodified lines | ||
| 55 | 55 | { |
| 56 | 56 | struct clocksource_mmio *cs; |
| 57 | 57 | |
| 58 | − if (bits > 32 || bits < 16) | |
| 58 | + if (bits > 64 || bits < 16) | |
| 59 | 59 | return -EINVAL; |
| 60 | 60 | |
| 61 | 61 | cs = kzalloc(sizeof(struct clocksource_mmio), GFP_KERNEL); |
| @@ -202,7 +202,7 @@ config ARM_SA1110_CPUFREQ · 201 unmodified lines | ||
| 202 | 202 | |
| 203 | 203 | config ARM_SCPI_CPUFREQ |
| 204 | 204 | tristate "SCPI based CPUfreq driver" |
| 205 | − depends on ARM_BIG_LITTLE_CPUFREQ && ARM_SCPI_PROTOCOL | |
| 205 | + depends on ARM_BIG_LITTLE_CPUFREQ && ARM_SCPI_PROTOCOL && COMMON_CLK_SCPI | |
| 206 | 206 | help |
| 207 | 207 | This adds the CPUfreq driver support for ARM big.LITTLE platforms |
| 208 | 208 | using SCPI protocol for CPU power management. |
| @@ -98,10 +98,11 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) · 97 unmodified lines | ||
| 98 | 98 | policy->max = cpu->perf_caps.highest_perf; |
| 99 | 99 | policy->cpuinfo.min_freq = policy->min; |
| 100 | 100 | policy->cpuinfo.max_freq = policy->max; |
| 101 | + policy->shared_type = cpu->shared_type; | |
| 101 | 102 | |
| 102 | 103 | if (policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) |
| 103 | 104 | cpumask_copy(policy->cpus, cpu->shared_cpu_map); |
| 104 | − else { | |
| 105 | + else if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL) { | |
| 105 | 106 | /* Support only SW_ANY for now. */ |
| 106 | 107 | pr_debug("Unsupported CPU co-ord type\n"); |
| 107 | 108 | return -EFAULT; |
| @@ -976,10 +976,14 @@ static int cpufreq_init_policy(struct cpufreq_policy *policy) · 975 unmodified lines | ||
| 976 | 976 | |
| 977 | 977 | new_policy.governor = gov; |
| 978 | 978 | |
| 979 | − /* Use the default policy if its valid. */ | |
| 980 | − if (cpufreq_driver->setpolicy) | |
| 981 | − cpufreq_parse_governor(gov->name, &new_policy.policy, NULL); | |
| 982 | − | |
| 979 | + /* Use the default policy if there is no last_policy. */ | |
| 980 | + if (cpufreq_driver->setpolicy) { | |
| 981 | + if (policy->last_policy) | |
| 982 | + new_policy.policy = policy->last_policy; | |
| 983 | + else | |
| 984 | + cpufreq_parse_governor(gov->name, &new_policy.policy, | |
| 985 | + NULL); | |
| 986 | + } | |
| 983 | 987 | /* set default policy */ |
| 984 | 988 | return cpufreq_set_policy(policy, &new_policy); |
| 985 | 989 | } |
| @@ -1330,6 +1334,8 @@ static void cpufreq_offline_prepare(unsigned int cpu) · 344 unmodified lines | ||
| 1330 | 1334 | if (has_target()) |
| 1331 | 1335 | strncpy(policy->last_governor, policy->governor->name, |
| 1332 | 1336 | CPUFREQ_NAME_LEN); |
| 1337 | + else | |
| 1338 | + policy->last_policy = policy->policy; | |
| 1333 | 1339 | } else if (cpu == policy->cpu) { |
| 1334 | 1340 | /* Nominate new CPU */ |
| 1335 | 1341 | policy->cpu = cpumask_any(policy->cpus); |
| @@ -1401,13 +1407,10 @@ static void cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif) · 65 unmodified lines | ||
| 1401 | 1407 | } |
| 1402 | 1408 | |
| 1403 | 1409 | cpumask_clear_cpu(cpu, policy->real_cpus); |
| 1410 | + remove_cpu_dev_symlink(policy, cpu); | |
| 1404 | 1411 | |
| 1405 | − if (cpumask_empty(policy->real_cpus)) { | |
| 1412 | + if (cpumask_empty(policy->real_cpus)) | |
| 1406 | 1413 | cpufreq_policy_free(policy, true); |
| 1407 | − return; | |
| 1408 | − } | |
| 1409 | − | |
| 1410 | − remove_cpu_dev_symlink(policy, cpu); | |
| 1411 | 1414 | } |
| 1412 | 1415 | |
| 1413 | 1416 | static void handle_update(struct work_struct *work) |
| @@ -1101,6 +1101,8 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy) · 1100 unmodified lines | ||
| 1101 | 1101 | policy->max >= policy->cpuinfo.max_freq) { |
| 1102 | 1102 | pr_debug("intel_pstate: set performance\n"); |
| 1103 | 1103 | limits = &performance_limits; |
| 1104 | + if (hwp_active) | |
| 1105 | + intel_pstate_hwp_set(); | |
| 1104 | 1106 | return 0; |
| 1105 | 1107 | } |
| 1106 | 1108 | |
| @@ -1108,7 +1110,8 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy) · 1 unmodified line | ||
| 1108 | 1110 | limits = &powersave_limits; |
| 1109 | 1111 | limits->min_policy_pct = (policy->min * 100) / policy->cpuinfo.max_freq; |
| 1110 | 1112 | limits->min_policy_pct = clamp_t(int, limits->min_policy_pct, 0 , 100); |
| 1111 | − limits->max_policy_pct = (policy->max * 100) / policy->cpuinfo.max_freq; | |
| 1113 | + limits->max_policy_pct = DIV_ROUND_UP(policy->max * 100, | |
| 1114 | + policy->cpuinfo.max_freq); | |
| 1112 | 1115 | limits->max_policy_pct = clamp_t(int, limits->max_policy_pct, 0 , 100); |
| 1113 | 1116 | |
| 1114 | 1117 | /* Normalize user input to [min_policy_pct, max_policy_pct] */ |
| @@ -1120,6 +1123,7 @@ static int intel_pstate_set_policy(struct cpufreq_policy *policy) · 5 unmodified lines | ||
| 1120 | 1123 | limits->max_sysfs_pct); |
| 1121 | 1124 | limits->max_perf_pct = max(limits->min_policy_pct, |
| 1122 | 1125 | limits->max_perf_pct); |
| 1126 | + limits->max_perf = round_up(limits->max_perf, 8); | |
| 1123 | 1127 | |
| 1124 | 1128 | /* Make sure min_perf_pct <= max_perf_pct */ |
| 1125 | 1129 | limits->min_perf_pct = min(limits->max_perf_pct, limits->min_perf_pct); |
| @@ -648,7 +648,7 @@ late_initcall(s3c_cpufreq_initcall); · 647 unmodified lines | ||
| 648 | 648 | * |
| 649 | 649 | * Register the given set of PLLs with the system. |
| 650 | 650 | */ |
| 651 | −int __init s3c_plltab_register(struct cpufreq_frequency_table *plls, | |
| 651 | +int s3c_plltab_register(struct cpufreq_frequency_table *plls, | |
| 652 | 652 | unsigned int plls_no) |
| 653 | 653 | { |
| 654 | 654 | struct cpufreq_frequency_table *vals; |
| @@ -409,7 +409,7 @@ static int ccm_nx_decrypt(struct aead_request *req, · 408 unmodified lines | ||
| 409 | 409 | processed += to_process; |
| 410 | 410 | } while (processed < nbytes); |
| 411 | 411 | |
| 412 | − rc = memcmp(csbcpb->cpb.aes_ccm.out_pat_or_mac, priv->oauth_tag, | |
| 412 | + rc = crypto_memneq(csbcpb->cpb.aes_ccm.out_pat_or_mac, priv->oauth_tag, | |
| 413 | 413 | authsize) ? -EBADMSG : 0; |
| 414 | 414 | out: |
| 415 | 415 | spin_unlock_irqrestore(&nx_ctx->lock, irq_flags); |
| @@ -21,6 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | |
| 22 | 22 | #include <crypto/internal/aead.h> |
| 23 | 23 | #include <crypto/aes.h> |
| 24 | +#include <crypto/algapi.h> | |
| 24 | 25 | #include <crypto/scatterwalk.h> |
| 25 | 26 | #include <linux/module.h> |
| 26 | 27 | #include <linux/types.h> |
| @@ -418,7 +419,7 @@ mac: · 391 unmodified lines | ||
| 418 | 419 | itag, req->src, req->assoclen + nbytes, |
| 419 | 420 | crypto_aead_authsize(crypto_aead_reqtfm(req)), |
| 420 | 421 | SCATTERWALK_FROM_SG); |
| 421 | − rc = memcmp(itag, otag, | |
| 422 | + rc = crypto_memneq(itag, otag, | |
| 422 | 423 | crypto_aead_authsize(crypto_aead_reqtfm(req))) ? |
| 423 | 424 | -EBADMSG : 0; |
| 424 | 425 | } |
| @@ -977,7 +977,7 @@ static void ipsec_esp_decrypt_swauth_done(struct device *dev, · 976 unmodified lines | ||
| 977 | 977 | } else |
| 978 | 978 | oicv = (char *)&edesc->link_tbl[0]; |
| 979 | 979 | |
| 980 | − err = memcmp(oicv, icv, authsize) ? -EBADMSG : 0; | |
| 980 | + err = crypto_memneq(oicv, icv, authsize) ? -EBADMSG : 0; | |
| 981 | 981 | } |
| 982 | 982 | |
| 983 | 983 | kfree(edesc); |
| @@ -122,12 +122,10 @@ int fpga_mgr_firmware_load(struct fpga_manager *mgr, u32 flags, · 121 unmodified lines | ||
| 122 | 122 | } |
| 123 | 123 | |
| 124 | 124 | ret = fpga_mgr_buf_load(mgr, flags, fw->data, fw->size); |
| 125 | − if (ret) | |
| 126 | − return ret; | |
| 127 | 125 | |
| 128 | 126 | release_firmware(fw); |
| 129 | 127 | |
| 130 | − return 0; | |
| 128 | + return ret; | |
| 131 | 129 | } |
| 132 | 130 | EXPORT_SYMBOL_GPL(fpga_mgr_firmware_load); |
| 133 | 131 | |
| @@ -256,7 +254,6 @@ int fpga_mgr_register(struct device *dev, const char *name, · 122 unmodified lines | ||
| 256 | 254 | void *priv) |
| 257 | 255 | { |
| 258 | 256 | struct fpga_manager *mgr; |
| 259 | − const char *dt_label; | |
| 260 | 257 | int id, ret; |
| 261 | 258 | |
| 262 | 259 | if (!mops || !mops->write_init || !mops->write || |
| @@ -300,11 +297,9 @@ int fpga_mgr_register(struct device *dev, const char *name, · 37 unmodified lines | ||
| 300 | 297 | mgr->dev.id = id; |
| 301 | 298 | dev_set_drvdata(dev, mgr); |
| 302 | 299 | |
| 303 | − dt_label = of_get_property(mgr->dev.of_node, "label", NULL); | |
| 304 | − if (dt_label) | |
| 305 | − ret = dev_set_name(&mgr->dev, "%s", dt_label); | |
| 306 | − else | |
| 307 | − ret = dev_set_name(&mgr->dev, "fpga%d", id); | |
| 300 | + ret = dev_set_name(&mgr->dev, "fpga%d", id); | |
| 301 | + if (ret) | |
| 302 | + goto error_device; | |
| 308 | 303 | |
| 309 | 304 | ret = device_add(&mgr->dev); |
| 310 | 305 | if (ret) |
| @@ -113,13 +113,16 @@ static int mmio_74xx_dir_out(struct gpio_chip *gc, unsigned int gpio, int val) · 112 unmodified lines | ||
| 113 | 113 | |
| 114 | 114 | static int mmio_74xx_gpio_probe(struct platform_device *pdev) |
| 115 | 115 | { |
| 116 | − const struct of_device_id *of_id = | |
| 117 | − of_match_device(mmio_74xx_gpio_ids, &pdev->dev); | |
| 116 | + const struct of_device_id *of_id; | |
| 118 | 117 | struct mmio_74xx_gpio_priv *priv; |
| 119 | 118 | struct resource *res; |
| 120 | 119 | void __iomem *dat; |
| 121 | 120 | int err; |
| 122 | 121 | |
| 122 | + of_id = of_match_device(mmio_74xx_gpio_ids, &pdev->dev); | |
| 123 | + if (!of_id) | |
| 124 | + return -ENODEV; | |
| 125 | + | |
| 123 | 126 | priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); |
| 124 | 127 | if (!priv) |
| 125 | 128 | return -ENOMEM; |
| @@ -1122,8 +1122,6 @@ static int omap_gpio_chip_init(struct gpio_bank *bank, struct irq_chip *irqc) · 1121 unmodified lines | ||
| 1122 | 1122 | /* MPUIO is a bit different, reading IRQ status clears it */ |
| 1123 | 1123 | if (bank->is_mpuio) { |
| 1124 | 1124 | irqc->irq_ack = dummy_irq_chip.irq_ack; |
| 1125 | − irqc->irq_mask = irq_gc_mask_set_bit; | |
| 1126 | − irqc->irq_unmask = irq_gc_mask_clr_bit; | |
| 1127 | 1125 | if (!bank->regs->wkup_en) |
| 1128 | 1126 | irqc->irq_set_wake = NULL; |
| 1129 | 1127 | } |
| @@ -167,6 +167,8 @@ static int palmas_gpio_probe(struct platform_device *pdev) · 166 unmodified lines | ||
| 167 | 167 | const struct palmas_device_data *dev_data; |
| 168 | 168 | |
| 169 | 169 | match = of_match_device(of_palmas_gpio_match, &pdev->dev); |
| 170 | + if (!match) | |
| 171 | + return -ENODEV; | |
| 170 | 172 | dev_data = match->data; |
| 171 | 173 | if (!dev_data) |
| 172 | 174 | dev_data = &palmas_dev_data; |
| @@ -187,11 +187,15 @@ MODULE_DEVICE_TABLE(of, syscon_gpio_ids); · 186 unmodified lines | ||
| 187 | 187 | static int syscon_gpio_probe(struct platform_device *pdev) |
| 188 | 188 | { |
| 189 | 189 | struct device *dev = &pdev->dev; |
| 190 | − const struct of_device_id *of_id = of_match_device(syscon_gpio_ids, dev); | |
| 190 | + const struct of_device_id *of_id; | |
| 191 | 191 | struct syscon_gpio_priv *priv; |
| 192 | 192 | struct device_node *np = dev->of_node; |
| 193 | 193 | int ret; |
| 194 | 194 | |
| 195 | + of_id = of_match_device(syscon_gpio_ids, dev); | |
| 196 | + if (!of_id) | |
| 197 | + return -ENODEV; | |
| 198 | + | |
| 195 | 199 | priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL); |
| 196 | 200 | if (!priv) |
| 197 | 201 | return -ENOMEM; |
| @@ -375,6 +375,60 @@ static int tegra_gpio_irq_set_wake(struct irq_data *d, unsigned int enable) · 374 unmodified lines | ||
| 375 | 375 | } |
| 376 | 376 | #endif |
| 377 | 377 | |
| 378 | +#ifdef CONFIG_DEBUG_FS | |
| 379 | + | |
| 380 | +#include <linux/debugfs.h> | |
| 381 | +#include <linux/seq_file.h> | |
| 382 | + | |
| 383 | +static int dbg_gpio_show(struct seq_file *s, void *unused) | |
| 384 | +{ | |
| 385 | + int i; | |
| 386 | + int j; | |
| 387 | + | |
| 388 | + for (i = 0; i < tegra_gpio_bank_count; i++) { | |
| 389 | + for (j = 0; j < 4; j++) { | |
| 390 | + int gpio = tegra_gpio_compose(i, j, 0); | |
| 391 | + seq_printf(s, | |
| 392 | + "%d:%d %02x %02x %02x %02x %02x %02x %06x\n", | |
| 393 | + i, j, | |
| 394 | + tegra_gpio_readl(GPIO_CNF(gpio)), | |
| 395 | + tegra_gpio_readl(GPIO_OE(gpio)), | |
| 396 | + tegra_gpio_readl(GPIO_OUT(gpio)), | |
| 397 | + tegra_gpio_readl(GPIO_IN(gpio)), | |
| 398 | + tegra_gpio_readl(GPIO_INT_STA(gpio)), | |
| 399 | + tegra_gpio_readl(GPIO_INT_ENB(gpio)), | |
| 400 | + tegra_gpio_readl(GPIO_INT_LVL(gpio))); | |
| 401 | + } | |
| 402 | + } | |
| 403 | + return 0; | |
| 404 | +} | |
| 405 | + | |
| 406 | +static int dbg_gpio_open(struct inode *inode, struct file *file) | |
| 407 | +{ | |
| 408 | + return single_open(file, dbg_gpio_show, &inode->i_private); | |
| 409 | +} | |
| 410 | + | |
| 411 | +static const struct file_operations debug_fops = { | |
| 412 | + .open = dbg_gpio_open, | |
| 413 | + .read = seq_read, | |
| 414 | + .llseek = seq_lseek, | |
| 415 | + .release = single_release, | |
| 416 | +}; | |
| 417 | + | |
| 418 | +static void tegra_gpio_debuginit(void) | |
| 419 | +{ | |
| 420 | + (void) debugfs_create_file("tegra_gpio", S_IRUGO, | |
| 421 | + NULL, NULL, &debug_fops); | |
| 422 | +} | |
| 423 | + | |
| 424 | +#else | |
| 425 | + | |
| 426 | +static inline void tegra_gpio_debuginit(void) | |
| 427 | +{ | |
| 428 | +} | |
| 429 | + | |
| 430 | +#endif | |
| 431 | + | |
| 378 | 432 | static struct irq_chip tegra_gpio_irq_chip = { |
| 379 | 433 | .name = "GPIO", |
| 380 | 434 | .irq_ack = tegra_gpio_irq_ack, |
| @@ -519,6 +573,8 @@ static int tegra_gpio_probe(struct platform_device *pdev) · 138 unmodified lines | ||
| 519 | 573 | spin_lock_init(&bank->lvl_lock[j]); |
| 520 | 574 | } |
| 521 | 575 | |
| 576 | + tegra_gpio_debuginit(); | |
| 577 | + | |
| 522 | 578 | return 0; |
| 523 | 579 | } |
| 524 | 580 | |
| @@ -536,52 +592,3 @@ static int __init tegra_gpio_init(void) · 11 unmodified lines | ||
| 536 | 592 | return platform_driver_register(&tegra_gpio_driver); |
| 537 | 593 | } |
| 538 | 594 | postcore_initcall(tegra_gpio_init); |
| 539 | − | |
| 540 | −#ifdef CONFIG_DEBUG_FS | |
| 541 | − | |
| 542 | −#include <linux/debugfs.h> | |
| 543 | −#include <linux/seq_file.h> | |
| 544 | − | |
| 545 | −static int dbg_gpio_show(struct seq_file *s, void *unused) | |
| 546 | −{ | |
| 547 | − int i; | |
| 548 | − int j; | |
| 549 | − | |
| 550 | − for (i = 0; i < tegra_gpio_bank_count; i++) { | |
| 551 | − for (j = 0; j < 4; j++) { | |
| 552 | − int gpio = tegra_gpio_compose(i, j, 0); | |
| 553 | − seq_printf(s, | |
| 554 | − "%d:%d %02x %02x %02x %02x %02x %02x %06x\n", | |
| 555 | − i, j, | |
| 556 | − tegra_gpio_readl(GPIO_CNF(gpio)), | |
| 557 | − tegra_gpio_readl(GPIO_OE(gpio)), | |
| 558 | − tegra_gpio_readl(GPIO_OUT(gpio)), | |
| 559 | − tegra_gpio_readl(GPIO_IN(gpio)), | |
| 560 | − tegra_gpio_readl(GPIO_INT_STA(gpio)), | |
| 561 | − tegra_gpio_readl(GPIO_INT_ENB(gpio)), | |
| 562 | − tegra_gpio_readl(GPIO_INT_LVL(gpio))); | |
| 563 | − } | |
| 564 | − } | |
| 565 | − return 0; | |
| 566 | −} | |
| 567 | − | |
| 568 | −static int dbg_gpio_open(struct inode *inode, struct file *file) | |
| 569 | −{ | |
| 570 | − return single_open(file, dbg_gpio_show, &inode->i_private); | |
| 571 | −} | |
| 572 | − | |
| 573 | −static const struct file_operations debug_fops = { | |
| 574 | − .open = dbg_gpio_open, | |
| 575 | − .read = seq_read, | |
| 576 | − .llseek = seq_lseek, | |
| 577 | − .release = single_release, | |
| 578 | −}; | |
| 579 | − | |
| 580 | −static int __init tegra_gpio_debuginit(void) | |
| 581 | −{ | |
| 582 | − (void) debugfs_create_file("tegra_gpio", S_IRUGO, | |
| 583 | − NULL, NULL, &debug_fops); | |
| 584 | − return 0; | |
| 585 | −} | |
| 586 | −late_initcall(tegra_gpio_debuginit); | |
| 587 | −#endif | |
| @@ -233,7 +233,7 @@ static struct gpio_desc *gpio_name_to_desc(const char * const name) · 232 unmodified lines | ||
| 233 | 233 | for (i = 0; i != chip->ngpio; ++i) { |
| 234 | 234 | struct gpio_desc *gpio = &chip->desc[i]; |
| 235 | 235 | |
| 236 | − if (!gpio->name) | |
| 236 | + if (!gpio->name || !name) | |
| 237 | 237 | continue; |
| 238 | 238 | |
| 239 | 239 | if (!strcmp(gpio->name, name)) { |
| @@ -496,6 +496,7 @@ struct amdgpu_bo_va_mapping { · 495 unmodified lines | ||
| 496 | 496 | |
| 497 | 497 | /* bo virtual addresses in a specific vm */ |
| 498 | 498 | struct amdgpu_bo_va { |
| 499 | + struct mutex mutex; | |
| 499 | 500 | /* protected by bo being reserved */ |
| 500 | 501 | struct list_head bo_list; |
| 501 | 502 | struct fence *last_pt_update; |
| @@ -538,6 +539,7 @@ struct amdgpu_bo { · 36 unmodified lines | ||
| 538 | 539 | /* Constant after initialization */ |
| 539 | 540 | struct amdgpu_device *adev; |
| 540 | 541 | struct drm_gem_object gem_base; |
| 542 | + struct amdgpu_bo *parent; | |
| 541 | 543 | |
| 542 | 544 | struct ttm_bo_kmap_obj dma_buf_vmap; |
| 543 | 545 | pid_t pid; |
| @@ -928,8 +930,6 @@ struct amdgpu_vm_id { · 384 unmodified lines | ||
| 928 | 930 | }; |
| 929 | 931 | |
| 930 | 932 | struct amdgpu_vm { |
| 931 | − struct mutex mutex; | |
| 932 | − | |
| 933 | 933 | struct rb_root va; |
| 934 | 934 | |
| 935 | 935 | /* protecting invalidated */ |
| @@ -956,6 +956,8 @@ struct amdgpu_vm { · 20 unmodified lines | ||
| 956 | 956 | struct amdgpu_vm_id ids[AMDGPU_MAX_RINGS]; |
| 957 | 957 | /* for interval tree */ |
| 958 | 958 | spinlock_t it_lock; |
| 959 | + /* protecting freed */ | |
| 960 | + spinlock_t freed_lock; | |
| 959 | 961 | }; |
| 960 | 962 | |
| 961 | 963 | struct amdgpu_vm_manager { |
| @@ -222,6 +222,8 @@ int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data) · 221 unmodified lines | ||
| 222 | 222 | } |
| 223 | 223 | |
| 224 | 224 | p->uf.bo = gem_to_amdgpu_bo(gobj); |
| 225 | + amdgpu_bo_ref(p->uf.bo); | |
| 226 | + drm_gem_object_unreference_unlocked(gobj); | |
| 225 | 227 | p->uf.offset = fence_data->offset; |
| 226 | 228 | } else { |
| 227 | 229 | ret = -EINVAL; |
| @@ -487,7 +489,7 @@ static void amdgpu_cs_parser_fini(struct amdgpu_cs_parser *parser, int error, bo · 259 unmodified lines | ||
| 487 | 489 | amdgpu_ib_free(parser->adev, &parser->ibs[i]); |
| 488 | 490 | kfree(parser->ibs); |
| 489 | 491 | if (parser->uf.bo) |
| 490 | − drm_gem_object_unreference_unlocked(&parser->uf.bo->gem_base); | |
| 492 | + amdgpu_bo_unref(&parser->uf.bo); | |
| 491 | 493 | } |
| 492 | 494 | |
| 493 | 495 | static int amdgpu_bo_vm_update_pte(struct amdgpu_cs_parser *p, |
| @@ -776,7 +778,7 @@ static int amdgpu_cs_free_job(struct amdgpu_job *job) · 282 unmodified lines | ||
| 776 | 778 | amdgpu_ib_free(job->adev, &job->ibs[i]); |
| 777 | 779 | kfree(job->ibs); |
| 778 | 780 | if (job->uf.bo) |
| 779 | − drm_gem_object_unreference_unlocked(&job->uf.bo->gem_base); | |
| 781 | + amdgpu_bo_unref(&job->uf.bo); | |
| 780 | 782 | return 0; |
| 781 | 783 | } |
| 782 | 784 | |
| @@ -784,8 +786,6 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · 1 unmodified line | ||
| 784 | 786 | { |
| 785 | 787 | struct amdgpu_device *adev = dev->dev_private; |
| 786 | 788 | union drm_amdgpu_cs *cs = data; |
| 787 | − struct amdgpu_fpriv *fpriv = filp->driver_priv; | |
| 788 | − struct amdgpu_vm *vm = &fpriv->vm; | |
| 789 | 789 | struct amdgpu_cs_parser parser = {}; |
| 790 | 790 | bool reserved_buffers = false; |
| 791 | 791 | int i, r; |
| @@ -803,7 +803,6 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · 11 unmodified lines | ||
| 803 | 803 | r = amdgpu_cs_handle_lockup(adev, r); |
| 804 | 804 | return r; |
| 805 | 805 | } |
| 806 | − mutex_lock(&vm->mutex); | |
| 807 | 806 | r = amdgpu_cs_parser_relocs(&parser); |
| 808 | 807 | if (r == -ENOMEM) |
| 809 | 808 | DRM_ERROR("Not enough memory for command submission!\n"); |
| @@ -888,7 +887,6 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · 78 unmodified lines | ||
| 888 | 887 | |
| 889 | 888 | out: |
| 890 | 889 | amdgpu_cs_parser_fini(&parser, r, reserved_buffers); |
| 891 | − mutex_unlock(&vm->mutex); | |
| 892 | 890 | r = amdgpu_cs_handle_lockup(adev, r); |
| 893 | 891 | return r; |
| 894 | 892 | } |
| @@ -73,6 +73,8 @@ static void amdgpu_flip_work_func(struct work_struct *__work) · 72 unmodified lines | ||
| 73 | 73 | struct drm_crtc *crtc = &amdgpuCrtc->base; |
| 74 | 74 | unsigned long flags; |
| 75 | 75 | unsigned i; |
| 76 | + int vpos, hpos, stat, min_udelay; | |
| 77 | + struct drm_vblank_crtc *vblank = &crtc->dev->vblank[work->crtc_id]; | |
| 76 | 78 | |
| 77 | 79 | amdgpu_flip_wait_fence(adev, &work->excl); |
| 78 | 80 | for (i = 0; i < work->shared_count; ++i) |
| @@ -81,6 +83,41 @@ static void amdgpu_flip_work_func(struct work_struct *__work) · 2 unmodified lines | ||
| 81 | 83 | /* We borrow the event spin lock for protecting flip_status */ |
| 82 | 84 | spin_lock_irqsave(&crtc->dev->event_lock, flags); |
| 83 | 85 | |
| 86 | + /* If this happens to execute within the "virtually extended" vblank | |
| 87 | + * interval before the start of the real vblank interval then it needs | |
| 88 | + * to delay programming the mmio flip until the real vblank is entered. | |
| 89 | + * This prevents completing a flip too early due to the way we fudge | |
| 90 | + * our vblank counter and vblank timestamps in order to work around the | |
| 91 | + * problem that the hw fires vblank interrupts before actual start of | |
| 92 | + * vblank (when line buffer refilling is done for a frame). It | |
| 93 | + * complements the fudging logic in amdgpu_get_crtc_scanoutpos() for | |
| 94 | + * timestamping and amdgpu_get_vblank_counter_kms() for vblank counts. | |
| 95 | + * | |
| 96 | + * In practice this won't execute very often unless on very fast | |
| 97 | + * machines because the time window for this to happen is very small. | |
| 98 | + */ | |
| 99 | + for (;;) { | |
| 100 | + /* GET_DISTANCE_TO_VBLANKSTART returns distance to real vblank | |
| 101 | + * start in hpos, and to the "fudged earlier" vblank start in | |
| 102 | + * vpos. | |
| 103 | + */ | |
| 104 | + stat = amdgpu_get_crtc_scanoutpos(adev->ddev, work->crtc_id, | |
| 105 | + GET_DISTANCE_TO_VBLANKSTART, | |
| 106 | + &vpos, &hpos, NULL, NULL, | |
| 107 | + &crtc->hwmode); | |
| 108 | + | |
| 109 | + if ((stat & (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE)) != | |
| 110 | + (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE) || | |
| 111 | + !(vpos >= 0 && hpos <= 0)) | |
| 112 | + break; | |
| 113 | + | |
| 114 | + /* Sleep at least until estimated real start of hw vblank */ | |
| 115 | + spin_unlock_irqrestore(&crtc->dev->event_lock, flags); | |
| 116 | + min_udelay = (-hpos + 1) * max(vblank->linedur_ns / 1000, 5); | |
| 117 | + usleep_range(min_udelay, 2 * min_udelay); | |
| 118 | + spin_lock_irqsave(&crtc->dev->event_lock, flags); | |
| 119 | + }; | |
| 120 | + | |
| 84 | 121 | /* do the flip (mmio) */ |
| 85 | 122 | adev->mode_info.funcs->page_flip(adev, work->crtc_id, work->base); |
| 86 | 123 | /* set the flip status */ |
| @@ -109,7 +146,7 @@ static void amdgpu_unpin_work_func(struct work_struct *__work) · 22 unmodified lines | ||
| 109 | 146 | } else |
| 110 | 147 | DRM_ERROR("failed to reserve buffer after flip\n"); |
| 111 | 148 | |
| 112 | − drm_gem_object_unreference_unlocked(&work->old_rbo->gem_base); | |
| 149 | + amdgpu_bo_unref(&work->old_rbo); | |
| 113 | 150 | kfree(work->shared); |
| 114 | 151 | kfree(work); |
| 115 | 152 | } |
| @@ -148,8 +185,8 @@ int amdgpu_crtc_page_flip(struct drm_crtc *crtc, · 32 unmodified lines | ||
| 148 | 185 | obj = old_amdgpu_fb->obj; |
| 149 | 186 | |
| 150 | 187 | /* take a reference to the old object */ |
| 151 | − drm_gem_object_reference(obj); | |
| 152 | 188 | work->old_rbo = gem_to_amdgpu_bo(obj); |
| 189 | + amdgpu_bo_ref(work->old_rbo); | |
| 153 | 190 | |
| 154 | 191 | new_amdgpu_fb = to_amdgpu_framebuffer(fb); |
| 155 | 192 | obj = new_amdgpu_fb->obj; |
| @@ -222,7 +259,7 @@ pflip_cleanup: · 66 unmodified lines | ||
| 222 | 259 | amdgpu_bo_unreserve(new_rbo); |
| 223 | 260 | |
| 224 | 261 | cleanup: |
| 225 | − drm_gem_object_unreference_unlocked(&work->old_rbo->gem_base); | |
| 262 | + amdgpu_bo_unref(&work->old_rbo); | |
| 226 | 263 | fence_put(work->excl); |
| 227 | 264 | for (i = 0; i < work->shared_count; ++i) |
| 228 | 265 | fence_put(work->shared[i]); |
| @@ -712,6 +749,15 @@ bool amdgpu_crtc_scaling_mode_fixup(struct drm_crtc *crtc, · 483 unmodified lines | ||
| 712 | 749 | * \param dev Device to query. |
| 713 | 750 | * \param pipe Crtc to query. |
| 714 | 751 | * \param flags Flags from caller (DRM_CALLED_FROM_VBLIRQ or 0). |
| 752 | + * For driver internal use only also supports these flags: | |
| 753 | + * | |
| 754 | + * USE_REAL_VBLANKSTART to use the real start of vblank instead | |
| 755 | + * of a fudged earlier start of vblank. | |
| 756 | + * | |
| 757 | + * GET_DISTANCE_TO_VBLANKSTART to return distance to the | |
| 758 | + * fudged earlier start of vblank in *vpos and the distance | |
| 759 | + * to true start of vblank in *hpos. | |
| 760 | + * | |
| 715 | 761 | * \param *vpos Location where vertical scanout position should be stored. |
| 716 | 762 | * \param *hpos Location where horizontal scanout position should go. |
| 717 | 763 | * \param *stime Target location for timestamp taken immediately before |
| @@ -776,10 +822,40 @@ int amdgpu_get_crtc_scanoutpos(struct drm_device *dev, unsigned int pipe, · 58 unmodified lines | ||
| 776 | 822 | vbl_end = 0; |
| 777 | 823 | } |
| 778 | 824 | |
| 825 | + /* Called from driver internal vblank counter query code? */ | |
| 826 | + if (flags & GET_DISTANCE_TO_VBLANKSTART) { | |
| 827 | + /* Caller wants distance from real vbl_start in *hpos */ | |
| 828 | + *hpos = *vpos - vbl_start; | |
| 829 | + } | |
| 830 | + | |
| 831 | + /* Fudge vblank to start a few scanlines earlier to handle the | |
| 832 | + * problem that vblank irqs fire a few scanlines before start | |
| 833 | + * of vblank. Some driver internal callers need the true vblank | |
| 834 | + * start to be used and signal this via the USE_REAL_VBLANKSTART flag. | |
| 835 | + * | |
| 836 | + * The cause of the "early" vblank irq is that the irq is triggered | |
| 837 | + * by the line buffer logic when the line buffer read position enters | |
| 838 | + * the vblank, whereas our crtc scanout position naturally lags the | |
| 839 | + * line buffer read position. | |
| 840 | + */ | |
| 841 | + if (!(flags & USE_REAL_VBLANKSTART)) | |
| 842 | + vbl_start -= adev->mode_info.crtcs[pipe]->lb_vblank_lead_lines; | |
| 843 | + | |
| 779 | 844 | /* Test scanout position against vblank region. */ |
| 780 | 845 | if ((*vpos < vbl_start) && (*vpos >= vbl_end)) |
| 781 | 846 | in_vbl = false; |
| 782 | 847 | |
| 848 | + /* In vblank? */ | |
| 849 | + if (in_vbl) | |
| 850 | + ret |= DRM_SCANOUTPOS_IN_VBLANK; | |
| 851 | + | |
| 852 | + /* Called from driver internal vblank counter query code? */ | |
| 853 | + if (flags & GET_DISTANCE_TO_VBLANKSTART) { | |
| 854 | + /* Caller wants distance from fudged earlier vbl_start */ | |
| 855 | + *vpos -= vbl_start; | |
| 856 | + return ret; | |
| 857 | + } | |
| 858 | + | |
| 783 | 859 | /* Check if inside vblank area and apply corrective offsets: |
| 784 | 860 | * vpos will then be >=0 in video scanout area, but negative |
| 785 | 861 | * within vblank area, counting down the number of lines until |
| @@ -795,32 +871,6 @@ int amdgpu_get_crtc_scanoutpos(struct drm_device *dev, unsigned int pipe, · 9 unmodified lines | ||
| 795 | 871 | /* Correct for shifted end of vbl at vbl_end. */ |
| 796 | 872 | *vpos = *vpos - vbl_end; |
| 797 | 873 | |
| 798 | − /* In vblank? */ | |
| 799 | − if (in_vbl) | |
| 800 | − ret |= DRM_SCANOUTPOS_IN_VBLANK; | |
| 801 | − | |
| 802 | − /* Is vpos outside nominal vblank area, but less than | |
| 803 | − * 1/100 of a frame height away from start of vblank? | |
| 804 | − * If so, assume this isn't a massively delayed vblank | |
| 805 | − * interrupt, but a vblank interrupt that fired a few | |
| 806 | − * microseconds before true start of vblank. Compensate | |
| 807 | − * by adding a full frame duration to the final timestamp. | |
| 808 | − * Happens, e.g., on ATI R500, R600. | |
| 809 | − * | |
| 810 | − * We only do this if DRM_CALLED_FROM_VBLIRQ. | |
| 811 | − */ | |
| 812 | − if ((flags & DRM_CALLED_FROM_VBLIRQ) && !in_vbl) { | |
| 813 | − vbl_start = mode->crtc_vdisplay; | |
| 814 | − vtotal = mode->crtc_vtotal; | |
| 815 | − | |
| 816 | − if (vbl_start - *vpos < vtotal / 100) { | |
| 817 | − *vpos -= vtotal; | |
| 818 | − | |
| 819 | − /* Signal this correction as "applied". */ | |
| 820 | − ret |= 0x8; | |
| 821 | − } | |
| 822 | − } | |
| 823 | − | |
| 824 | 874 | return ret; |
| 825 | 875 | } |
| 826 | 876 | |
| @@ -115,12 +115,9 @@ int amdgpu_gem_object_open(struct drm_gem_object *obj, struct drm_file *file_pri · 114 unmodified lines | ||
| 115 | 115 | struct amdgpu_vm *vm = &fpriv->vm; |
| 116 | 116 | struct amdgpu_bo_va *bo_va; |
| 117 | 117 | int r; |
| 118 | − mutex_lock(&vm->mutex); | |
| 119 | 118 | r = amdgpu_bo_reserve(rbo, false); |
| 120 | − if (r) { | |
| 121 | − mutex_unlock(&vm->mutex); | |
| 119 | + if (r) | |
| 122 | 120 | return r; |
| 123 | − } | |
| 124 | 121 | |
| 125 | 122 | bo_va = amdgpu_vm_bo_find(vm, rbo); |
| 126 | 123 | if (!bo_va) { |
| @@ -129,7 +126,6 @@ int amdgpu_gem_object_open(struct drm_gem_object *obj, struct drm_file *file_pri · 2 unmodified lines | ||
| 129 | 126 | ++bo_va->ref_count; |
| 130 | 127 | } |
| 131 | 128 | amdgpu_bo_unreserve(rbo); |
| 132 | − mutex_unlock(&vm->mutex); | |
| 133 | 129 | return 0; |
| 134 | 130 | } |
| 135 | 131 | |
| @@ -142,10 +138,8 @@ void amdgpu_gem_object_close(struct drm_gem_object *obj, · 6 unmodified lines | ||
| 142 | 138 | struct amdgpu_vm *vm = &fpriv->vm; |
| 143 | 139 | struct amdgpu_bo_va *bo_va; |
| 144 | 140 | int r; |
| 145 | − mutex_lock(&vm->mutex); | |
| 146 | 141 | r = amdgpu_bo_reserve(rbo, true); |
| 147 | 142 | if (r) { |
| 148 | − mutex_unlock(&vm->mutex); | |
| 149 | 143 | dev_err(adev->dev, "leaking bo va because " |
| 150 | 144 | "we fail to reserve bo (%d)\n", r); |
| 151 | 145 | return; |
| @@ -157,7 +151,6 @@ void amdgpu_gem_object_close(struct drm_gem_object *obj, · 5 unmodified lines | ||
| 157 | 151 | } |
| 158 | 152 | } |
| 159 | 153 | amdgpu_bo_unreserve(rbo); |
| 160 | − mutex_unlock(&vm->mutex); | |
| 161 | 154 | } |
| 162 | 155 | |
| 163 | 156 | static int amdgpu_gem_handle_lockup(struct amdgpu_device *adev, int r) |
| @@ -242,8 +235,9 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, · 78 unmodified lines | ||
| 242 | 235 | AMDGPU_GEM_USERPTR_REGISTER)) |
| 243 | 236 | return -EINVAL; |
| 244 | 237 | |
| 245 | − if (!(args->flags & AMDGPU_GEM_USERPTR_ANONONLY) || | |
| 246 | − !(args->flags & AMDGPU_GEM_USERPTR_REGISTER)) { | |
| 238 | + if (!(args->flags & AMDGPU_GEM_USERPTR_READONLY) && ( | |
| 239 | + !(args->flags & AMDGPU_GEM_USERPTR_ANONONLY) || | |
| 240 | + !(args->flags & AMDGPU_GEM_USERPTR_REGISTER))) { | |
| 247 | 241 | |
| 248 | 242 | /* if we want to write to it we must require anonymous |
| 249 | 243 | memory and install a MMU notifier */ |
| @@ -483,6 +477,14 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 233 unmodified lines | ||
| 483 | 477 | if (domain == AMDGPU_GEM_DOMAIN_CPU) |
| 484 | 478 | goto error_unreserve; |
| 485 | 479 | } |
| 480 | + list_for_each_entry(entry, &duplicates, head) { | |
| 481 | + domain = amdgpu_mem_type_to_domain(entry->bo->mem.mem_type); | |
| 482 | + /* if anything is swapped out don't swap it in here, | |
| 483 | + just abort and wait for the next CS */ | |
| 484 | + if (domain == AMDGPU_GEM_DOMAIN_CPU) | |
| 485 | + goto error_unreserve; | |
| 486 | + } | |
| 487 | + | |
| 486 | 488 | r = amdgpu_vm_update_page_directory(adev, bo_va->vm); |
| 487 | 489 | if (r) |
| 488 | 490 | goto error_unreserve; |
| @@ -553,7 +555,6 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 64 unmodified lines | ||
| 553 | 555 | gobj = drm_gem_object_lookup(dev, filp, args->handle); |
| 554 | 556 | if (gobj == NULL) |
| 555 | 557 | return -ENOENT; |
| 556 | − mutex_lock(&fpriv->vm.mutex); | |
| 557 | 558 | rbo = gem_to_amdgpu_bo(gobj); |
| 558 | 559 | INIT_LIST_HEAD(&list); |
| 559 | 560 | INIT_LIST_HEAD(&duplicates); |
| @@ -568,7 +569,6 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 8 unmodified lines | ||
| 568 | 569 | } |
| 569 | 570 | r = ttm_eu_reserve_buffers(&ticket, &list, true, &duplicates); |
| 570 | 571 | if (r) { |
| 571 | − mutex_unlock(&fpriv->vm.mutex); | |
| 572 | 572 | drm_gem_object_unreference_unlocked(gobj); |
| 573 | 573 | return r; |
| 574 | 574 | } |
| @@ -577,7 +577,6 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 2 unmodified lines | ||
| 577 | 577 | if (!bo_va) { |
| 578 | 578 | ttm_eu_backoff_reservation(&ticket, &list); |
| 579 | 579 | drm_gem_object_unreference_unlocked(gobj); |
| 580 | − mutex_unlock(&fpriv->vm.mutex); | |
| 581 | 580 | return -ENOENT; |
| 582 | 581 | } |
| 583 | 582 | |
| @@ -602,7 +601,7 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 18 unmodified lines | ||
| 602 | 601 | ttm_eu_backoff_reservation(&ticket, &list); |
| 603 | 602 | if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE)) |
| 604 | 603 | amdgpu_gem_va_update_vm(adev, bo_va, args->operation); |
| 605 | − mutex_unlock(&fpriv->vm.mutex); | |
| 604 | + | |
| 606 | 605 | drm_gem_object_unreference_unlocked(gobj); |
| 607 | 606 | return r; |
| 608 | 607 | } |
| @@ -611,13 +611,59 @@ void amdgpu_driver_preclose_kms(struct drm_device *dev, · 610 unmodified lines | ||
| 611 | 611 | u32 amdgpu_get_vblank_counter_kms(struct drm_device *dev, unsigned int pipe) |
| 612 | 612 | { |
| 613 | 613 | struct amdgpu_device *adev = dev->dev_private; |
| 614 | + int vpos, hpos, stat; | |
| 615 | + u32 count; | |
| 614 | 616 | |
| 615 | 617 | if (pipe >= adev->mode_info.num_crtc) { |
| 616 | 618 | DRM_ERROR("Invalid crtc %u\n", pipe); |
| 617 | 619 | return -EINVAL; |
| 618 | 620 | } |
| 619 | 621 | |
| 620 | − return amdgpu_display_vblank_get_counter(adev, pipe); | |
| 622 | + /* The hw increments its frame counter at start of vsync, not at start | |
| 623 | + * of vblank, as is required by DRM core vblank counter handling. | |
| 624 | + * Cook the hw count here to make it appear to the caller as if it | |
| 625 | + * incremented at start of vblank. We measure distance to start of | |
| 626 | + * vblank in vpos. vpos therefore will be >= 0 between start of vblank | |
| 627 | + * and start of vsync, so vpos >= 0 means to bump the hw frame counter | |
| 628 | + * result by 1 to give the proper appearance to caller. | |
| 629 | + */ | |
| 630 | + if (adev->mode_info.crtcs[pipe]) { | |
| 631 | + /* Repeat readout if needed to provide stable result if | |
| 632 | + * we cross start of vsync during the queries. | |
| 633 | + */ | |
| 634 | + do { | |
| 635 | + count = amdgpu_display_vblank_get_counter(adev, pipe); | |
| 636 | + /* Ask amdgpu_get_crtc_scanoutpos to return vpos as | |
| 637 | + * distance to start of vblank, instead of regular | |
| 638 | + * vertical scanout pos. | |
| 639 | + */ | |
| 640 | + stat = amdgpu_get_crtc_scanoutpos( | |
| 641 | + dev, pipe, GET_DISTANCE_TO_VBLANKSTART, | |
| 642 | + &vpos, &hpos, NULL, NULL, | |
| 643 | + &adev->mode_info.crtcs[pipe]->base.hwmode); | |
| 644 | + } while (count != amdgpu_display_vblank_get_counter(adev, pipe)); | |
| 645 | + | |
| 646 | + if (((stat & (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE)) != | |
| 647 | + (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE))) { | |
| 648 | + DRM_DEBUG_VBL("Query failed! stat %d\n", stat); | |
| 649 | + } else { | |
| 650 | + DRM_DEBUG_VBL("crtc %d: dist from vblank start %d\n", | |
| 651 | + pipe, vpos); | |
| 652 | + | |
| 653 | + /* Bump counter if we are at >= leading edge of vblank, | |
| 654 | + * but before vsync where vpos would turn negative and | |
| 655 | + * the hw counter really increments. | |
| 656 | + */ | |
| 657 | + if (vpos >= 0) | |
| 658 | + count++; | |
| 659 | + } | |
| 660 | + } else { | |
| 661 | + /* Fallback to use value as is. */ | |
| 662 | + count = amdgpu_display_vblank_get_counter(adev, pipe); | |
| 663 | + DRM_DEBUG_VBL("NULL mode info! Returned count may be wrong.\n"); | |
| 664 | + } | |
| 665 | + | |
| 666 | + return count; | |
| 621 | 667 | } |
| 622 | 668 | |
| 623 | 669 | /** |
| @@ -407,6 +407,7 @@ struct amdgpu_crtc { · 406 unmodified lines | ||
| 407 | 407 | u32 line_time; |
| 408 | 408 | u32 wm_low; |
| 409 | 409 | u32 wm_high; |
| 410 | + u32 lb_vblank_lead_lines; | |
| 410 | 411 | struct drm_display_mode hw_mode; |
| 411 | 412 | }; |
| 412 | 413 | |
| @@ -528,6 +529,10 @@ struct amdgpu_framebuffer { · 115 unmodified lines | ||
| 528 | 529 | #define ENCODER_MODE_IS_DP(em) (((em) == ATOM_ENCODER_MODE_DP) || \ |
| 529 | 530 | ((em) == ATOM_ENCODER_MODE_DP_MST)) |
| 530 | 531 | |
| 532 | +/* Driver internal use only flags of amdgpu_get_crtc_scanoutpos() */ | |
| 533 | +#define USE_REAL_VBLANKSTART (1 << 30) | |
| 534 | +#define GET_DISTANCE_TO_VBLANKSTART (1 << 31) | |
| 535 | + | |
| 531 | 536 | void amdgpu_link_encoder_connector(struct drm_device *dev); |
| 532 | 537 | |
| 533 | 538 | struct drm_connector * |
| @@ -100,6 +100,7 @@ static void amdgpu_ttm_bo_destroy(struct ttm_buffer_object *tbo) · 99 unmodified lines | ||
| 100 | 100 | list_del_init(&bo->list); |
| 101 | 101 | mutex_unlock(&bo->adev->gem.mutex); |
| 102 | 102 | drm_gem_object_release(&bo->gem_base); |
| 103 | + amdgpu_bo_unref(&bo->parent); | |
| 103 | 104 | kfree(bo->metadata); |
| 104 | 105 | kfree(bo); |
| 105 | 106 | } |
| @@ -587,9 +587,13 @@ static int amdgpu_ttm_backend_bind(struct ttm_tt *ttm, · 586 unmodified lines | ||
| 587 | 587 | uint32_t flags = amdgpu_ttm_tt_pte_flags(gtt->adev, ttm, bo_mem); |
| 588 | 588 | int r; |
| 589 | 589 | |
| 590 | − if (gtt->userptr) | |
| 591 | − amdgpu_ttm_tt_pin_userptr(ttm); | |
| 592 | − | |
| 590 | + if (gtt->userptr) { | |
| 591 | + r = amdgpu_ttm_tt_pin_userptr(ttm); | |
| 592 | + if (r) { | |
| 593 | + DRM_ERROR("failed to pin userptr\n"); | |
| 594 | + return r; | |
| 595 | + } | |
| 596 | + } | |
| 593 | 597 | gtt->offset = (unsigned long)(bo_mem->start << PAGE_SHIFT); |
| 594 | 598 | if (!ttm->num_pages) { |
| 595 | 599 | WARN(1, "nothing to bind %lu pages for mreg %p back %p!\n", |
| @@ -797,11 +801,12 @@ uint32_t amdgpu_ttm_tt_pte_flags(struct amdgpu_device *adev, struct ttm_tt *ttm, · 201 unmodified lines | ||
| 797 | 801 | if (mem && mem->mem_type != TTM_PL_SYSTEM) |
| 798 | 802 | flags |= AMDGPU_PTE_VALID; |
| 799 | 803 | |
| 800 | − if (mem && mem->mem_type == TTM_PL_TT) | |
| 804 | + if (mem && mem->mem_type == TTM_PL_TT) { | |
| 801 | 805 | flags |= AMDGPU_PTE_SYSTEM; |
| 802 | 806 | |
| 803 | − if (!ttm || ttm->caching_state == tt_cached) | |
| 804 | − flags |= AMDGPU_PTE_SNOOPED; | |
| 807 | + if (ttm->caching_state == tt_cached) | |
| 808 | + flags |= AMDGPU_PTE_SNOOPED; | |
| 809 | + } | |
| 805 | 810 | |
| 806 | 811 | if (adev->asic_type >= CHIP_TOPAZ) |
| 807 | 812 | flags |= AMDGPU_PTE_EXECUTABLE; |
| @@ -392,7 +392,10 @@ int amdgpu_vce_get_create_msg(struct amdgpu_ring *ring, uint32_t handle, · 391 unmodified lines | ||
| 392 | 392 | ib->ptr[ib->length_dw++] = 0x00000001; /* session cmd */ |
| 393 | 393 | ib->ptr[ib->length_dw++] = handle; |
| 394 | 394 | |
| 395 | − ib->ptr[ib->length_dw++] = 0x00000030; /* len */ | |
| 395 | + if ((ring->adev->vce.fw_version >> 24) >= 52) | |
| 396 | + ib->ptr[ib->length_dw++] = 0x00000040; /* len */ | |
| 397 | + else | |
| 398 | + ib->ptr[ib->length_dw++] = 0x00000030; /* len */ | |
| 396 | 399 | ib->ptr[ib->length_dw++] = 0x01000001; /* create cmd */ |
| 397 | 400 | ib->ptr[ib->length_dw++] = 0x00000000; |
| 398 | 401 | ib->ptr[ib->length_dw++] = 0x00000042; |
| @@ -404,6 +407,12 @@ int amdgpu_vce_get_create_msg(struct amdgpu_ring *ring, uint32_t handle, · 5 unmodified lines | ||
| 404 | 407 | ib->ptr[ib->length_dw++] = 0x00000100; |
| 405 | 408 | ib->ptr[ib->length_dw++] = 0x0000000c; |
| 406 | 409 | ib->ptr[ib->length_dw++] = 0x00000000; |
| 410 | + if ((ring->adev->vce.fw_version >> 24) >= 52) { | |
| 411 | + ib->ptr[ib->length_dw++] = 0x00000000; | |
| 412 | + ib->ptr[ib->length_dw++] = 0x00000000; | |
| 413 | + ib->ptr[ib->length_dw++] = 0x00000000; | |
| 414 | + ib->ptr[ib->length_dw++] = 0x00000000; | |
| 415 | + } | |
| 407 | 416 | |
| 408 | 417 | ib->ptr[ib->length_dw++] = 0x00000014; /* len */ |
| 409 | 418 | ib->ptr[ib->length_dw++] = 0x05000005; /* feedback buffer */ |
| @@ -885,17 +885,21 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev, · 884 unmodified lines | ||
| 885 | 885 | struct amdgpu_bo_va_mapping *mapping; |
| 886 | 886 | int r; |
| 887 | 887 | |
| 888 | + spin_lock(&vm->freed_lock); | |
| 888 | 889 | while (!list_empty(&vm->freed)) { |
| 889 | 890 | mapping = list_first_entry(&vm->freed, |
| 890 | 891 | struct amdgpu_bo_va_mapping, list); |
| 891 | 892 | list_del(&mapping->list); |
| 892 | − | |
| 893 | + spin_unlock(&vm->freed_lock); | |
| 893 | 894 | r = amdgpu_vm_bo_update_mapping(adev, vm, mapping, 0, 0, NULL); |
| 894 | 895 | kfree(mapping); |
| 895 | 896 | if (r) |
| 896 | 897 | return r; |
| 897 | 898 | |
| 899 | + spin_lock(&vm->freed_lock); | |
| 898 | 900 | } |
| 901 | + spin_unlock(&vm->freed_lock); | |
| 902 | + | |
| 899 | 903 | return 0; |
| 900 | 904 | |
| 901 | 905 | } |
| @@ -922,8 +926,9 @@ int amdgpu_vm_clear_invalids(struct amdgpu_device *adev, · 20 unmodified lines | ||
| 922 | 926 | bo_va = list_first_entry(&vm->invalidated, |
| 923 | 927 | struct amdgpu_bo_va, vm_status); |
| 924 | 928 | spin_unlock(&vm->status_lock); |
| 925 | − | |
| 929 | + mutex_lock(&bo_va->mutex); | |
| 926 | 930 | r = amdgpu_vm_bo_update(adev, bo_va, NULL); |
| 931 | + mutex_unlock(&bo_va->mutex); | |
| 927 | 932 | if (r) |
| 928 | 933 | return r; |
| 929 | 934 | |
| @@ -967,7 +972,7 @@ struct amdgpu_bo_va *amdgpu_vm_bo_add(struct amdgpu_device *adev, · 37 unmodified lines | ||
| 967 | 972 | INIT_LIST_HEAD(&bo_va->valids); |
| 968 | 973 | INIT_LIST_HEAD(&bo_va->invalids); |
| 969 | 974 | INIT_LIST_HEAD(&bo_va->vm_status); |
| 970 | − | |
| 975 | + mutex_init(&bo_va->mutex); | |
| 971 | 976 | list_add_tail(&bo_va->bo_list, &bo->va); |
| 972 | 977 | |
| 973 | 978 | return bo_va; |
| @@ -1045,7 +1050,9 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev, · 71 unmodified lines | ||
| 1045 | 1050 | mapping->offset = offset; |
| 1046 | 1051 | mapping->flags = flags; |
| 1047 | 1052 | |
| 1053 | + mutex_lock(&bo_va->mutex); | |
| 1048 | 1054 | list_add(&mapping->list, &bo_va->invalids); |
| 1055 | + mutex_unlock(&bo_va->mutex); | |
| 1049 | 1056 | spin_lock(&vm->it_lock); |
| 1050 | 1057 | interval_tree_insert(&mapping->it, &vm->va); |
| 1051 | 1058 | spin_unlock(&vm->it_lock); |
| @@ -1076,6 +1083,11 @@ int amdgpu_vm_bo_map(struct amdgpu_device *adev, · 24 unmodified lines | ||
| 1076 | 1083 | if (r) |
| 1077 | 1084 | goto error_free; |
| 1078 | 1085 | |
| 1086 | + /* Keep a reference to the page table to avoid freeing | |
| 1087 | + * them up in the wrong order. | |
| 1088 | + */ | |
| 1089 | + pt->parent = amdgpu_bo_ref(vm->page_directory); | |
| 1090 | + | |
| 1079 | 1091 | r = amdgpu_vm_clear_bo(adev, pt); |
| 1080 | 1092 | if (r) { |
| 1081 | 1093 | amdgpu_bo_unref(&pt); |
| @@ -1121,7 +1133,7 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, · 39 unmodified lines | ||
| 1121 | 1133 | bool valid = true; |
| 1122 | 1134 | |
| 1123 | 1135 | saddr /= AMDGPU_GPU_PAGE_SIZE; |
| 1124 | − | |
| 1136 | + mutex_lock(&bo_va->mutex); | |
| 1125 | 1137 | list_for_each_entry(mapping, &bo_va->valids, list) { |
| 1126 | 1138 | if (mapping->it.start == saddr) |
| 1127 | 1139 | break; |
| @@ -1135,20 +1147,25 @@ int amdgpu_vm_bo_unmap(struct amdgpu_device *adev, · 7 unmodified lines | ||
| 1135 | 1147 | break; |
| 1136 | 1148 | } |
| 1137 | 1149 | |
| 1138 | − if (&mapping->list == &bo_va->invalids) | |
| 1150 | + if (&mapping->list == &bo_va->invalids) { | |
| 1151 | + mutex_unlock(&bo_va->mutex); | |
| 1139 | 1152 | return -ENOENT; |
| 1153 | + } | |
| 1140 | 1154 | } |
| 1141 | − | |
| 1155 | + mutex_unlock(&bo_va->mutex); | |
| 1142 | 1156 | list_del(&mapping->list); |
| 1143 | 1157 | spin_lock(&vm->it_lock); |
| 1144 | 1158 | interval_tree_remove(&mapping->it, &vm->va); |
| 1145 | 1159 | spin_unlock(&vm->it_lock); |
| 1146 | 1160 | trace_amdgpu_vm_bo_unmap(bo_va, mapping); |
| 1147 | 1161 | |
| 1148 | − if (valid) | |
| 1162 | + if (valid) { | |
| 1163 | + spin_lock(&vm->freed_lock); | |
| 1149 | 1164 | list_add(&mapping->list, &vm->freed); |
| 1150 | − else | |
| 1165 | + spin_unlock(&vm->freed_lock); | |
| 1166 | + } else { | |
| 1151 | 1167 | kfree(mapping); |
| 1168 | + } | |
| 1152 | 1169 | |
| 1153 | 1170 | return 0; |
| 1154 | 1171 | } |
| @@ -1181,7 +1198,9 @@ void amdgpu_vm_bo_rmv(struct amdgpu_device *adev, · 26 unmodified lines | ||
| 1181 | 1198 | interval_tree_remove(&mapping->it, &vm->va); |
| 1182 | 1199 | spin_unlock(&vm->it_lock); |
| 1183 | 1200 | trace_amdgpu_vm_bo_unmap(bo_va, mapping); |
| 1201 | + spin_lock(&vm->freed_lock); | |
| 1184 | 1202 | list_add(&mapping->list, &vm->freed); |
| 1203 | + spin_unlock(&vm->freed_lock); | |
| 1185 | 1204 | } |
| 1186 | 1205 | list_for_each_entry_safe(mapping, next, &bo_va->invalids, list) { |
| 1187 | 1206 | list_del(&mapping->list); |
| @@ -1190,8 +1209,8 @@ void amdgpu_vm_bo_rmv(struct amdgpu_device *adev, · 2 unmodified lines | ||
| 1190 | 1209 | spin_unlock(&vm->it_lock); |
| 1191 | 1210 | kfree(mapping); |
| 1192 | 1211 | } |
| 1193 | − | |
| 1194 | 1212 | fence_put(bo_va->last_pt_update); |
| 1213 | + mutex_destroy(&bo_va->mutex); | |
| 1195 | 1214 | kfree(bo_va); |
| 1196 | 1215 | } |
| 1197 | 1216 | |
| @@ -1236,13 +1255,13 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm) · 38 unmodified lines | ||
| 1236 | 1255 | vm->ids[i].id = 0; |
| 1237 | 1256 | vm->ids[i].flushed_updates = NULL; |
| 1238 | 1257 | } |
| 1239 | − mutex_init(&vm->mutex); | |
| 1240 | 1258 | vm->va = RB_ROOT; |
| 1241 | 1259 | spin_lock_init(&vm->status_lock); |
| 1242 | 1260 | INIT_LIST_HEAD(&vm->invalidated); |
| 1243 | 1261 | INIT_LIST_HEAD(&vm->cleared); |
| 1244 | 1262 | INIT_LIST_HEAD(&vm->freed); |
| 1245 | 1263 | spin_lock_init(&vm->it_lock); |
| 1264 | + spin_lock_init(&vm->freed_lock); | |
| 1246 | 1265 | pd_size = amdgpu_vm_directory_size(adev); |
| 1247 | 1266 | pd_entries = amdgpu_vm_num_pdes(adev); |
| 1248 | 1267 | |
| @@ -1320,7 +1339,6 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) · 71 unmodified lines | ||
| 1320 | 1339 | fence_put(vm->ids[i].flushed_updates); |
| 1321 | 1340 | } |
| 1322 | 1341 | |
| 1323 | − mutex_destroy(&vm->mutex); | |
| 1324 | 1342 | } |
| 1325 | 1343 | |
| 1326 | 1344 | /** |
| @@ -1250,7 +1250,7 @@ static void dce_v10_0_program_watermarks(struct amdgpu_device *adev, · 1249 unmodified lines | ||
| 1250 | 1250 | u32 pixel_period; |
| 1251 | 1251 | u32 line_time = 0; |
| 1252 | 1252 | u32 latency_watermark_a = 0, latency_watermark_b = 0; |
| 1253 | − u32 tmp, wm_mask; | |
| 1253 | + u32 tmp, wm_mask, lb_vblank_lead_lines = 0; | |
| 1254 | 1254 | |
| 1255 | 1255 | if (amdgpu_crtc->base.enabled && num_heads && mode) { |
| 1256 | 1256 | pixel_period = 1000000 / (u32)mode->clock; |
| @@ -1333,6 +1333,7 @@ static void dce_v10_0_program_watermarks(struct amdgpu_device *adev, · 76 unmodified lines | ||
| 1333 | 1333 | (adev->mode_info.disp_priority == 2)) { |
| 1334 | 1334 | DRM_DEBUG_KMS("force priority to high\n"); |
| 1335 | 1335 | } |
| 1336 | + lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 1336 | 1337 | } |
| 1337 | 1338 | |
| 1338 | 1339 | /* select wm A */ |
| @@ -1357,6 +1358,8 @@ static void dce_v10_0_program_watermarks(struct amdgpu_device *adev, · 18 unmodified lines | ||
| 1357 | 1358 | amdgpu_crtc->line_time = line_time; |
| 1358 | 1359 | amdgpu_crtc->wm_high = latency_watermark_a; |
| 1359 | 1360 | amdgpu_crtc->wm_low = latency_watermark_b; |
| 1361 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 1362 | + amdgpu_crtc->lb_vblank_lead_lines = lb_vblank_lead_lines; | |
| 1360 | 1363 | } |
| 1361 | 1364 | |
| 1362 | 1365 | /** |
| @@ -1238,7 +1238,7 @@ static void dce_v11_0_program_watermarks(struct amdgpu_device *adev, · 1237 unmodified lines | ||
| 1238 | 1238 | u32 pixel_period; |
| 1239 | 1239 | u32 line_time = 0; |
| 1240 | 1240 | u32 latency_watermark_a = 0, latency_watermark_b = 0; |
| 1241 | − u32 tmp, wm_mask; | |
| 1241 | + u32 tmp, wm_mask, lb_vblank_lead_lines = 0; | |
| 1242 | 1242 | |
| 1243 | 1243 | if (amdgpu_crtc->base.enabled && num_heads && mode) { |
| 1244 | 1244 | pixel_period = 1000000 / (u32)mode->clock; |
| @@ -1321,6 +1321,7 @@ static void dce_v11_0_program_watermarks(struct amdgpu_device *adev, · 76 unmodified lines | ||
| 1321 | 1321 | (adev->mode_info.disp_priority == 2)) { |
| 1322 | 1322 | DRM_DEBUG_KMS("force priority to high\n"); |
| 1323 | 1323 | } |
| 1324 | + lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 1324 | 1325 | } |
| 1325 | 1326 | |
| 1326 | 1327 | /* select wm A */ |
| @@ -1345,6 +1346,8 @@ static void dce_v11_0_program_watermarks(struct amdgpu_device *adev, · 18 unmodified lines | ||
| 1345 | 1346 | amdgpu_crtc->line_time = line_time; |
| 1346 | 1347 | amdgpu_crtc->wm_high = latency_watermark_a; |
| 1347 | 1348 | amdgpu_crtc->wm_low = latency_watermark_b; |
| 1349 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 1350 | + amdgpu_crtc->lb_vblank_lead_lines = lb_vblank_lead_lines; | |
| 1348 | 1351 | } |
| 1349 | 1352 | |
| 1350 | 1353 | /** |
| @@ -1193,7 +1193,7 @@ static void dce_v8_0_program_watermarks(struct amdgpu_device *adev, · 1192 unmodified lines | ||
| 1193 | 1193 | u32 pixel_period; |
| 1194 | 1194 | u32 line_time = 0; |
| 1195 | 1195 | u32 latency_watermark_a = 0, latency_watermark_b = 0; |
| 1196 | − u32 tmp, wm_mask; | |
| 1196 | + u32 tmp, wm_mask, lb_vblank_lead_lines = 0; | |
| 1197 | 1197 | |
| 1198 | 1198 | if (amdgpu_crtc->base.enabled && num_heads && mode) { |
| 1199 | 1199 | pixel_period = 1000000 / (u32)mode->clock; |
| @@ -1276,6 +1276,7 @@ static void dce_v8_0_program_watermarks(struct amdgpu_device *adev, · 76 unmodified lines | ||
| 1276 | 1276 | (adev->mode_info.disp_priority == 2)) { |
| 1277 | 1277 | DRM_DEBUG_KMS("force priority to high\n"); |
| 1278 | 1278 | } |
| 1279 | + lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 1279 | 1280 | } |
| 1280 | 1281 | |
| 1281 | 1282 | /* select wm A */ |
| @@ -1302,6 +1303,8 @@ static void dce_v8_0_program_watermarks(struct amdgpu_device *adev, · 20 unmodified lines | ||
| 1302 | 1303 | amdgpu_crtc->line_time = line_time; |
| 1303 | 1304 | amdgpu_crtc->wm_high = latency_watermark_a; |
| 1304 | 1305 | amdgpu_crtc->wm_low = latency_watermark_b; |
| 1306 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 1307 | + amdgpu_crtc->lb_vblank_lead_lines = lb_vblank_lead_lines; | |
| 1305 | 1308 | } |
| 1306 | 1309 | |
| 1307 | 1310 | /** |
| @@ -513,7 +513,7 @@ static int gmc_v7_0_gart_enable(struct amdgpu_device *adev) · 512 unmodified lines | ||
| 513 | 513 | WREG32(mmVM_L2_CNTL3, tmp); |
| 514 | 514 | /* setup context0 */ |
| 515 | 515 | WREG32(mmVM_CONTEXT0_PAGE_TABLE_START_ADDR, adev->mc.gtt_start >> 12); |
| 516 | − WREG32(mmVM_CONTEXT0_PAGE_TABLE_END_ADDR, (adev->mc.gtt_end >> 12) - 1); | |
| 516 | + WREG32(mmVM_CONTEXT0_PAGE_TABLE_END_ADDR, adev->mc.gtt_end >> 12); | |
| 517 | 517 | WREG32(mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR, adev->gart.table_addr >> 12); |
| 518 | 518 | WREG32(mmVM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR, |
| 519 | 519 | (u32)(adev->dummy_page.addr >> 12)); |
| @@ -657,7 +657,7 @@ static int gmc_v8_0_gart_enable(struct amdgpu_device *adev) · 656 unmodified lines | ||
| 657 | 657 | WREG32(mmVM_L2_CNTL4, tmp); |
| 658 | 658 | /* setup context0 */ |
| 659 | 659 | WREG32(mmVM_CONTEXT0_PAGE_TABLE_START_ADDR, adev->mc.gtt_start >> 12); |
| 660 | − WREG32(mmVM_CONTEXT0_PAGE_TABLE_END_ADDR, (adev->mc.gtt_end >> 12) - 1); | |
| 660 | + WREG32(mmVM_CONTEXT0_PAGE_TABLE_END_ADDR, adev->mc.gtt_end >> 12); | |
| 661 | 661 | WREG32(mmVM_CONTEXT0_PAGE_TABLE_BASE_ADDR, adev->gart.table_addr >> 12); |
| 662 | 662 | WREG32(mmVM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR, |
| 663 | 663 | (u32)(adev->dummy_page.addr >> 12)); |
| @@ -40,6 +40,9 @@ · 39 unmodified lines | ||
| 40 | 40 | |
| 41 | 41 | #define GRBM_GFX_INDEX__VCE_INSTANCE__SHIFT 0x04 |
| 42 | 42 | #define GRBM_GFX_INDEX__VCE_INSTANCE_MASK 0x10 |
| 43 | +#define mmVCE_LMI_VCPU_CACHE_40BIT_BAR0 0x8616 | |
| 44 | +#define mmVCE_LMI_VCPU_CACHE_40BIT_BAR1 0x8617 | |
| 45 | +#define mmVCE_LMI_VCPU_CACHE_40BIT_BAR2 0x8618 | |
| 43 | 46 | |
| 44 | 47 | #define VCE_V3_0_FW_SIZE (384 * 1024) |
| 45 | 48 | #define VCE_V3_0_STACK_SIZE (64 * 1024) |
| @@ -130,9 +133,11 @@ static int vce_v3_0_start(struct amdgpu_device *adev) · 84 unmodified lines | ||
| 130 | 133 | |
| 131 | 134 | /* set BUSY flag */ |
| 132 | 135 | WREG32_P(mmVCE_STATUS, 1, ~1); |
| 133 | − | |
| 134 | − WREG32_P(mmVCE_VCPU_CNTL, VCE_VCPU_CNTL__CLK_EN_MASK, | |
| 135 | − ~VCE_VCPU_CNTL__CLK_EN_MASK); | |
| 136 | + if (adev->asic_type >= CHIP_STONEY) | |
| 137 | + WREG32_P(mmVCE_VCPU_CNTL, 1, ~0x200001); | |
| 138 | + else | |
| 139 | + WREG32_P(mmVCE_VCPU_CNTL, VCE_VCPU_CNTL__CLK_EN_MASK, | |
| 140 | + ~VCE_VCPU_CNTL__CLK_EN_MASK); | |
| 136 | 141 | |
| 137 | 142 | WREG32_P(mmVCE_SOFT_RESET, |
| 138 | 143 | VCE_SOFT_RESET__ECPU_SOFT_RESET_MASK, |
| @@ -391,8 +396,12 @@ static void vce_v3_0_mc_resume(struct amdgpu_device *adev, int idx) · 252 unmodified lines | ||
| 391 | 396 | WREG32(mmVCE_LMI_SWAP_CNTL, 0); |
| 392 | 397 | WREG32(mmVCE_LMI_SWAP_CNTL1, 0); |
| 393 | 398 | WREG32(mmVCE_LMI_VM_CTRL, 0); |
| 394 | − | |
| 395 | − WREG32(mmVCE_LMI_VCPU_CACHE_40BIT_BAR, (adev->vce.gpu_addr >> 8)); | |
| 399 | + if (adev->asic_type >= CHIP_STONEY) { | |
| 400 | + WREG32(mmVCE_LMI_VCPU_CACHE_40BIT_BAR0, (adev->vce.gpu_addr >> 8)); | |
| 401 | + WREG32(mmVCE_LMI_VCPU_CACHE_40BIT_BAR1, (adev->vce.gpu_addr >> 8)); | |
| 402 | + WREG32(mmVCE_LMI_VCPU_CACHE_40BIT_BAR2, (adev->vce.gpu_addr >> 8)); | |
| 403 | + } else | |
| 404 | + WREG32(mmVCE_LMI_VCPU_CACHE_40BIT_BAR, (adev->vce.gpu_addr >> 8)); | |
| 396 | 405 | offset = AMDGPU_VCE_FIRMWARE_OFFSET; |
| 397 | 406 | size = VCE_V3_0_FW_SIZE; |
| 398 | 407 | WREG32(mmVCE_VCPU_CACHE_OFFSET0, offset & 0x7fffffff); |
| @@ -576,6 +585,11 @@ static int vce_v3_0_process_interrupt(struct amdgpu_device *adev, · 177 unmodified lines | ||
| 576 | 585 | struct amdgpu_iv_entry *entry) |
| 577 | 586 | { |
| 578 | 587 | DRM_DEBUG("IH: VCE\n"); |
| 588 | + | |
| 589 | + WREG32_P(mmVCE_SYS_INT_STATUS, | |
| 590 | + VCE_SYS_INT_STATUS__VCE_SYS_INT_TRAP_INTERRUPT_INT_MASK, | |
| 591 | + ~VCE_SYS_INT_STATUS__VCE_SYS_INT_TRAP_INTERRUPT_INT_MASK); | |
| 592 | + | |
| 579 | 593 | switch (entry->src_data) { |
| 580 | 594 | case 0: |
| 581 | 595 | amdgpu_fence_process(&adev->vce.ring[0]); |
| @@ -30,8 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #define CREATE_TRACE_POINTS |
| 31 | 31 | #include "gpu_sched_trace.h" |
| 32 | 32 | |
| 33 | −static struct amd_sched_job * | |
| 34 | −amd_sched_entity_pop_job(struct amd_sched_entity *entity); | |
| 33 | +static bool amd_sched_entity_is_ready(struct amd_sched_entity *entity); | |
| 35 | 34 | static void amd_sched_wakeup(struct amd_gpu_scheduler *sched); |
| 36 | 35 | |
| 37 | 36 | struct kmem_cache *sched_fence_slab; |
| @@ -64,36 +63,36 @@ static void amd_sched_rq_remove_entity(struct amd_sched_rq *rq, · 26 unmodified lines | ||
| 64 | 63 | } |
| 65 | 64 | |
| 66 | 65 | /** |
| 67 | − * Select next job from a specified run queue with round robin policy. | |
| 68 | − * Return NULL if nothing available. | |
| 66 | + * Select an entity which could provide a job to run | |
| 67 | + * | |
| 68 | + * @rq The run queue to check. | |
| 69 | + * | |
| 70 | + * Try to find a ready entity, returns NULL if none found. | |
| 69 | 71 | */ |
| 70 | −static struct amd_sched_job * | |
| 71 | −amd_sched_rq_select_job(struct amd_sched_rq *rq) | |
| 72 | +static struct amd_sched_entity * | |
| 73 | +amd_sched_rq_select_entity(struct amd_sched_rq *rq) | |
| 72 | 74 | { |
| 73 | 75 | struct amd_sched_entity *entity; |
| 74 | − struct amd_sched_job *sched_job; | |
| 75 | 76 | |
| 76 | 77 | spin_lock(&rq->lock); |
| 77 | 78 | |
| 78 | 79 | entity = rq->current_entity; |
| 79 | 80 | if (entity) { |
| 80 | 81 | list_for_each_entry_continue(entity, &rq->entities, list) { |
| 81 | − sched_job = amd_sched_entity_pop_job(entity); | |
| 82 | − if (sched_job) { | |
| 82 | + if (amd_sched_entity_is_ready(entity)) { | |
| 83 | 83 | rq->current_entity = entity; |
| 84 | 84 | spin_unlock(&rq->lock); |
| 85 | − return sched_job; | |
| 85 | + return entity; | |
| 86 | 86 | } |
| 87 | 87 | } |
| 88 | 88 | } |
| 89 | 89 | |
| 90 | 90 | list_for_each_entry(entity, &rq->entities, list) { |
| 91 | 91 | |
| 92 | − sched_job = amd_sched_entity_pop_job(entity); | |
| 93 | − if (sched_job) { | |
| 92 | + if (amd_sched_entity_is_ready(entity)) { | |
| 94 | 93 | rq->current_entity = entity; |
| 95 | 94 | spin_unlock(&rq->lock); |
| 96 | − return sched_job; | |
| 95 | + return entity; | |
| 97 | 96 | } |
| 98 | 97 | |
| 99 | 98 | if (entity == rq->current_entity) |
| @@ -176,6 +175,24 @@ static bool amd_sched_entity_is_idle(struct amd_sched_entity *entity) · 76 unmodified lines | ||
| 176 | 175 | return false; |
| 177 | 176 | } |
| 178 | 177 | |
| 178 | +/** | |
| 179 | + * Check if entity is ready | |
| 180 | + * | |
| 181 | + * @entity The pointer to a valid scheduler entity | |
| 182 | + * | |
| 183 | + * Return true if entity could provide a job. | |
| 184 | + */ | |
| 185 | +static bool amd_sched_entity_is_ready(struct amd_sched_entity *entity) | |
| 186 | +{ | |
| 187 | + if (kfifo_is_empty(&entity->job_queue)) | |
| 188 | + return false; | |
| 189 | + | |
| 190 | + if (ACCESS_ONCE(entity->dependency)) | |
| 191 | + return false; | |
| 192 | + | |
| 193 | + return true; | |
| 194 | +} | |
| 195 | + | |
| 179 | 196 | /** |
| 180 | 197 | * Destroy a context entity |
| 181 | 198 | * |
| @@ -211,32 +228,53 @@ static void amd_sched_entity_wakeup(struct fence *f, struct fence_cb *cb) · 29 unmodified lines | ||
| 211 | 228 | amd_sched_wakeup(entity->sched); |
| 212 | 229 | } |
| 213 | 230 | |
| 231 | +static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) | |
| 232 | +{ | |
| 233 | + struct amd_gpu_scheduler *sched = entity->sched; | |
| 234 | + struct fence * fence = entity->dependency; | |
| 235 | + struct amd_sched_fence *s_fence; | |
| 236 | + | |
| 237 | + if (fence->context == entity->fence_context) { | |
| 238 | + /* We can ignore fences from ourself */ | |
| 239 | + fence_put(entity->dependency); | |
| 240 | + return false; | |
| 241 | + } | |
| 242 | + | |
| 243 | + s_fence = to_amd_sched_fence(fence); | |
| 244 | + if (s_fence && s_fence->sched == sched) { | |
| 245 | + /* Fence is from the same scheduler */ | |
| 246 | + if (test_bit(AMD_SCHED_FENCE_SCHEDULED_BIT, &fence->flags)) { | |
| 247 | + /* Ignore it when it is already scheduled */ | |
| 248 | + fence_put(entity->dependency); | |
| 249 | + return false; | |
| 250 | + } | |
| 251 | + | |
| 252 | + /* Wait for fence to be scheduled */ | |
| 253 | + entity->cb.func = amd_sched_entity_wakeup; | |
| 254 | + list_add_tail(&entity->cb.node, &s_fence->scheduled_cb); | |
| 255 | + return true; | |
| 256 | + } | |
| 257 | + | |
| 258 | + if (!fence_add_callback(entity->dependency, &entity->cb, | |
| 259 | + amd_sched_entity_wakeup)) | |
| 260 | + return true; | |
| 261 | + | |
| 262 | + fence_put(entity->dependency); | |
| 263 | + return false; | |
| 264 | +} | |
| 265 | + | |
| 214 | 266 | static struct amd_sched_job * |
| 215 | 267 | amd_sched_entity_pop_job(struct amd_sched_entity *entity) |
| 216 | 268 | { |
| 217 | 269 | struct amd_gpu_scheduler *sched = entity->sched; |
| 218 | 270 | struct amd_sched_job *sched_job; |
| 219 | 271 | |
| 220 | − if (ACCESS_ONCE(entity->dependency)) | |
| 221 | − return NULL; | |
| 222 | − | |
| 223 | 272 | if (!kfifo_out_peek(&entity->job_queue, &sched_job, sizeof(sched_job))) |
| 224 | 273 | return NULL; |
| 225 | 274 | |
| 226 | − while ((entity->dependency = sched->ops->dependency(sched_job))) { | |
| 227 | − | |
| 228 | − if (entity->dependency->context == entity->fence_context) { | |
| 229 | − /* We can ignore fences from ourself */ | |
| 230 | − fence_put(entity->dependency); | |
| 231 | − continue; | |
| 232 | − } | |
| 233 | − | |
| 234 | − if (fence_add_callback(entity->dependency, &entity->cb, | |
| 235 | − amd_sched_entity_wakeup)) | |
| 236 | − fence_put(entity->dependency); | |
| 237 | − else | |
| 275 | + while ((entity->dependency = sched->ops->dependency(sched_job))) | |
| 276 | + if (amd_sched_entity_add_dependency_cb(entity)) | |
| 238 | 277 | return NULL; |
| 239 | − } | |
| 240 | 278 | |
| 241 | 279 | return sched_job; |
| 242 | 280 | } |
| @@ -250,6 +288,7 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity) · 7 unmodified lines | ||
| 250 | 288 | */ |
| 251 | 289 | static bool amd_sched_entity_in(struct amd_sched_job *sched_job) |
| 252 | 290 | { |
| 291 | + struct amd_gpu_scheduler *sched = sched_job->sched; | |
| 253 | 292 | struct amd_sched_entity *entity = sched_job->s_entity; |
| 254 | 293 | bool added, first = false; |
| 255 | 294 | |
| @@ -264,7 +303,7 @@ static bool amd_sched_entity_in(struct amd_sched_job *sched_job) · 8 unmodified lines | ||
| 264 | 303 | |
| 265 | 304 | /* first job wakes up scheduler */ |
| 266 | 305 | if (first) |
| 267 | − amd_sched_wakeup(sched_job->sched); | |
| 306 | + amd_sched_wakeup(sched); | |
| 268 | 307 | |
| 269 | 308 | return added; |
| 270 | 309 | } |
| @@ -280,9 +319,9 @@ void amd_sched_entity_push_job(struct amd_sched_job *sched_job) · 9 unmodified lines | ||
| 280 | 319 | { |
| 281 | 320 | struct amd_sched_entity *entity = sched_job->s_entity; |
| 282 | 321 | |
| 322 | + trace_amd_sched_job(sched_job); | |
| 283 | 323 | wait_event(entity->sched->job_scheduled, |
| 284 | 324 | amd_sched_entity_in(sched_job)); |
| 285 | − trace_amd_sched_job(sched_job); | |
| 286 | 325 | } |
| 287 | 326 | |
| 288 | 327 | /** |
| @@ -304,22 +343,22 @@ static void amd_sched_wakeup(struct amd_gpu_scheduler *sched) · 15 unmodified lines | ||
| 304 | 343 | } |
| 305 | 344 | |
| 306 | 345 | /** |
| 307 | − * Select next to run | |
| 346 | + * Select next entity to process | |
| 308 | 347 | */ |
| 309 | −static struct amd_sched_job * | |
| 310 | −amd_sched_select_job(struct amd_gpu_scheduler *sched) | |
| 348 | +static struct amd_sched_entity * | |
| 349 | +amd_sched_select_entity(struct amd_gpu_scheduler *sched) | |
| 311 | 350 | { |
| 312 | − struct amd_sched_job *sched_job; | |
| 351 | + struct amd_sched_entity *entity; | |
| 313 | 352 | |
| 314 | 353 | if (!amd_sched_ready(sched)) |
| 315 | 354 | return NULL; |
| 316 | 355 | |
| 317 | 356 | /* Kernel run queue has higher priority than normal run queue*/ |
| 318 | − sched_job = amd_sched_rq_select_job(&sched->kernel_rq); | |
| 319 | − if (sched_job == NULL) | |
| 320 | − sched_job = amd_sched_rq_select_job(&sched->sched_rq); | |
| 357 | + entity = amd_sched_rq_select_entity(&sched->kernel_rq); | |
| 358 | + if (entity == NULL) | |
| 359 | + entity = amd_sched_rq_select_entity(&sched->sched_rq); | |
| 321 | 360 | |
| 322 | − return sched_job; | |
| 361 | + return entity; | |
| 323 | 362 | } |
| 324 | 363 | |
| 325 | 364 | static void amd_sched_process_job(struct fence *f, struct fence_cb *cb) |
| @@ -381,13 +420,16 @@ static int amd_sched_main(void *param) · 55 unmodified lines | ||
| 381 | 420 | unsigned long flags; |
| 382 | 421 | |
| 383 | 422 | wait_event_interruptible(sched->wake_up_worker, |
| 384 | − kthread_should_stop() || | |
| 385 | − (sched_job = amd_sched_select_job(sched))); | |
| 423 | + (entity = amd_sched_select_entity(sched)) || | |
| 424 | + kthread_should_stop()); | |
| 386 | 425 | |
| 426 | + if (!entity) | |
| 427 | + continue; | |
| 428 | + | |
| 429 | + sched_job = amd_sched_entity_pop_job(entity); | |
| 387 | 430 | if (!sched_job) |
| 388 | 431 | continue; |
| 389 | 432 | |
| 390 | − entity = sched_job->s_entity; | |
| 391 | 433 | s_fence = sched_job->s_fence; |
| 392 | 434 | |
| 393 | 435 | if (sched->timeout != MAX_SCHEDULE_TIMEOUT) { |
| @@ -400,6 +442,7 @@ static int amd_sched_main(void *param) · 6 unmodified lines | ||
| 400 | 442 | |
| 401 | 443 | atomic_inc(&sched->hw_rq_count); |
| 402 | 444 | fence = sched->ops->run_job(sched_job); |
| 445 | + amd_sched_fence_scheduled(s_fence); | |
| 403 | 446 | if (fence) { |
| 404 | 447 | r = fence_add_callback(fence, &s_fence->cb, |
| 405 | 448 | amd_sched_process_job); |
| @@ -27,6 +27,8 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <linux/kfifo.h> |
| 28 | 28 | #include <linux/fence.h> |
| 29 | 29 | |
| 30 | +#define AMD_SCHED_FENCE_SCHEDULED_BIT FENCE_FLAG_USER_BITS | |
| 31 | + | |
| 30 | 32 | struct amd_gpu_scheduler; |
| 31 | 33 | struct amd_sched_rq; |
| 32 | 34 | |
| @@ -68,6 +70,7 @@ struct amd_sched_rq { · 35 unmodified lines | ||
| 68 | 70 | struct amd_sched_fence { |
| 69 | 71 | struct fence base; |
| 70 | 72 | struct fence_cb cb; |
| 73 | + struct list_head scheduled_cb; | |
| 71 | 74 | struct amd_gpu_scheduler *sched; |
| 72 | 75 | spinlock_t lock; |
| 73 | 76 | void *owner; |
| @@ -134,7 +137,7 @@ void amd_sched_entity_push_job(struct amd_sched_job *sched_job); · 60 unmodified lines | ||
| 134 | 137 | |
| 135 | 138 | struct amd_sched_fence *amd_sched_fence_create( |
| 136 | 139 | struct amd_sched_entity *s_entity, void *owner); |
| 140 | +void amd_sched_fence_scheduled(struct amd_sched_fence *fence); | |
| 137 | 141 | void amd_sched_fence_signal(struct amd_sched_fence *fence); |
| 138 | 142 | |
| 139 | − | |
| 140 | 143 | #endif |
| @@ -35,6 +35,8 @@ struct amd_sched_fence *amd_sched_fence_create(struct amd_sched_entity *s_entity · 34 unmodified lines | ||
| 35 | 35 | fence = kmem_cache_zalloc(sched_fence_slab, GFP_KERNEL); |
| 36 | 36 | if (fence == NULL) |
| 37 | 37 | return NULL; |
| 38 | + | |
| 39 | + INIT_LIST_HEAD(&fence->scheduled_cb); | |
| 38 | 40 | fence->owner = owner; |
| 39 | 41 | fence->sched = s_entity->sched; |
| 40 | 42 | spin_lock_init(&fence->lock); |
| @@ -55,6 +57,17 @@ void amd_sched_fence_signal(struct amd_sched_fence *fence) · 14 unmodified lines | ||
| 55 | 57 | FENCE_TRACE(&fence->base, "was already signaled\n"); |
| 56 | 58 | } |
| 57 | 59 | |
| 60 | +void amd_sched_fence_scheduled(struct amd_sched_fence *s_fence) | |
| 61 | +{ | |
| 62 | + struct fence_cb *cur, *tmp; | |
| 63 | + | |
| 64 | + set_bit(AMD_SCHED_FENCE_SCHEDULED_BIT, &s_fence->base.flags); | |
| 65 | + list_for_each_entry_safe(cur, tmp, &s_fence->scheduled_cb, node) { | |
| 66 | + list_del_init(&cur->node); | |
| 67 | + cur->func(&s_fence->base, cur); | |
| 68 | + } | |
| 69 | +} | |
| 70 | + | |
| 58 | 71 | static const char *amd_sched_fence_get_driver_name(struct fence *fence) |
| 59 | 72 | { |
| 60 | 73 | return "amd_sched"; |
| @@ -160,6 +160,11 @@ int drm_setmaster_ioctl(struct drm_device *dev, void *data, · 159 unmodified lines | ||
| 160 | 160 | goto out_unlock; |
| 161 | 161 | } |
| 162 | 162 | |
| 163 | + if (!file_priv->allowed_master) { | |
| 164 | + ret = drm_new_set_master(dev, file_priv); | |
| 165 | + goto out_unlock; | |
| 166 | + } | |
| 167 | + | |
| 163 | 168 | file_priv->minor->master = drm_master_get(file_priv->master); |
| 164 | 169 | file_priv->is_master = 1; |
| 165 | 170 | if (dev->driver->master_set) { |
| @@ -125,6 +125,60 @@ static int drm_cpu_valid(void) · 124 unmodified lines | ||
| 125 | 125 | return 1; |
| 126 | 126 | } |
| 127 | 127 | |
| 128 | +/** | |
| 129 | + * drm_new_set_master - Allocate a new master object and become master for the | |
| 130 | + * associated master realm. | |
| 131 | + * | |
| 132 | + * @dev: The associated device. | |
| 133 | + * @fpriv: File private identifying the client. | |
| 134 | + * | |
| 135 | + * This function must be called with dev::struct_mutex held. | |
| 136 | + * Returns negative error code on failure. Zero on success. | |
| 137 | + */ | |
| 138 | +int drm_new_set_master(struct drm_device *dev, struct drm_file *fpriv) | |
| 139 | +{ | |
| 140 | + struct drm_master *old_master; | |
| 141 | + int ret; | |
| 142 | + | |
| 143 | + lockdep_assert_held_once(&dev->master_mutex); | |
| 144 | + | |
| 145 | + /* create a new master */ | |
| 146 | + fpriv->minor->master = drm_master_create(fpriv->minor); | |
| 147 | + if (!fpriv->minor->master) | |
| 148 | + return -ENOMEM; | |
| 149 | + | |
| 150 | + /* take another reference for the copy in the local file priv */ | |
| 151 | + old_master = fpriv->master; | |
| 152 | + fpriv->master = drm_master_get(fpriv->minor->master); | |
| 153 | + | |
| 154 | + if (dev->driver->master_create) { | |
| 155 | + ret = dev->driver->master_create(dev, fpriv->master); | |
| 156 | + if (ret) | |
| 157 | + goto out_err; | |
| 158 | + } | |
| 159 | + if (dev->driver->master_set) { | |
| 160 | + ret = dev->driver->master_set(dev, fpriv, true); | |
| 161 | + if (ret) | |
| 162 | + goto out_err; | |
| 163 | + } | |
| 164 | + | |
| 165 | + fpriv->is_master = 1; | |
| 166 | + fpriv->allowed_master = 1; | |
| 167 | + fpriv->authenticated = 1; | |
| 168 | + if (old_master) | |
| 169 | + drm_master_put(&old_master); | |
| 170 | + | |
| 171 | + return 0; | |
| 172 | + | |
| 173 | +out_err: | |
| 174 | + /* drop both references and restore old master on failure */ | |
| 175 | + drm_master_put(&fpriv->minor->master); | |
| 176 | + drm_master_put(&fpriv->master); | |
| 177 | + fpriv->master = old_master; | |
| 178 | + | |
| 179 | + return ret; | |
| 180 | +} | |
| 181 | + | |
| 128 | 182 | /** |
| 129 | 183 | * Called whenever a process opens /dev/drm. |
| 130 | 184 | * |
| @@ -189,35 +243,9 @@ static int drm_open_helper(struct file *filp, struct drm_minor *minor) · 58 unmodified lines | ||
| 189 | 243 | mutex_lock(&dev->master_mutex); |
| 190 | 244 | if (drm_is_primary_client(priv) && !priv->minor->master) { |
| 191 | 245 | /* create a new master */ |
| 192 | − priv->minor->master = drm_master_create(priv->minor); | |
| 193 | − if (!priv->minor->master) { | |
| 194 | − ret = -ENOMEM; | |
| 246 | + ret = drm_new_set_master(dev, priv); | |
| 247 | + if (ret) | |
| 195 | 248 | goto out_close; |
| 196 | − } | |
| 197 | − | |
| 198 | − priv->is_master = 1; | |
| 199 | − /* take another reference for the copy in the local file priv */ | |
| 200 | − priv->master = drm_master_get(priv->minor->master); | |
| 201 | − priv->authenticated = 1; | |
| 202 | − | |
| 203 | − if (dev->driver->master_create) { | |
| 204 | − ret = dev->driver->master_create(dev, priv->master); | |
| 205 | − if (ret) { | |
| 206 | − /* drop both references if this fails */ | |
| 207 | − drm_master_put(&priv->minor->master); | |
| 208 | − drm_master_put(&priv->master); | |
| 209 | − goto out_close; | |
| 210 | − } | |
| 211 | − } | |
| 212 | − if (dev->driver->master_set) { | |
| 213 | − ret = dev->driver->master_set(dev, priv, true); | |
| 214 | − if (ret) { | |
| 215 | − /* drop both references if this fails */ | |
| 216 | − drm_master_put(&priv->minor->master); | |
| 217 | − drm_master_put(&priv->master); | |
| 218 | − goto out_close; | |
| 219 | − } | |
| 220 | − } | |
| 221 | 249 | } else if (drm_is_primary_client(priv)) { |
| 222 | 250 | /* get a reference to the master */ |
| 223 | 251 | priv->master = drm_master_get(priv->minor->master); |
| @@ -980,7 +980,8 @@ static void send_vblank_event(struct drm_device *dev, · 979 unmodified lines | ||
| 980 | 980 | struct drm_pending_vblank_event *e, |
| 981 | 981 | unsigned long seq, struct timeval *now) |
| 982 | 982 | { |
| 983 | − WARN_ON_SMP(!spin_is_locked(&dev->event_lock)); | |
| 983 | + assert_spin_locked(&dev->event_lock); | |
| 984 | + | |
| 984 | 985 | e->event.sequence = seq; |
| 985 | 986 | e->event.tv_sec = now->tv_sec; |
| 986 | 987 | e->event.tv_usec = now->tv_usec; |
| @@ -992,6 +993,57 @@ static void send_vblank_event(struct drm_device *dev, · 5 unmodified lines | ||
| 992 | 993 | e->event.sequence); |
| 993 | 994 | } |
| 994 | 995 | |
| 996 | +/** | |
| 997 | + * drm_arm_vblank_event - arm vblank event after pageflip | |
| 998 | + * @dev: DRM device | |
| 999 | + * @pipe: CRTC index | |
| 1000 | + * @e: the event to prepare to send | |
| 1001 | + * | |
| 1002 | + * A lot of drivers need to generate vblank events for the very next vblank | |
| 1003 | + * interrupt. For example when the page flip interrupt happens when the page | |
| 1004 | + * flip gets armed, but not when it actually executes within the next vblank | |
| 1005 | + * period. This helper function implements exactly the required vblank arming | |
| 1006 | + * behaviour. | |
| 1007 | + * | |
| 1008 | + * Caller must hold event lock. Caller must also hold a vblank reference for | |
| 1009 | + * the event @e, which will be dropped when the next vblank arrives. | |
| 1010 | + * | |
| 1011 | + * This is the legacy version of drm_crtc_arm_vblank_event(). | |
| 1012 | + */ | |
| 1013 | +void drm_arm_vblank_event(struct drm_device *dev, unsigned int pipe, | |
| 1014 | + struct drm_pending_vblank_event *e) | |
| 1015 | +{ | |
| 1016 | + assert_spin_locked(&dev->event_lock); | |
| 1017 | + | |
| 1018 | + e->pipe = pipe; | |
| 1019 | + e->event.sequence = drm_vblank_count(dev, pipe); | |
| 1020 | + list_add_tail(&e->base.link, &dev->vblank_event_list); | |
| 1021 | +} | |
| 1022 | +EXPORT_SYMBOL(drm_arm_vblank_event); | |
| 1023 | + | |
| 1024 | +/** | |
| 1025 | + * drm_crtc_arm_vblank_event - arm vblank event after pageflip | |
| 1026 | + * @crtc: the source CRTC of the vblank event | |
| 1027 | + * @e: the event to send | |
| 1028 | + * | |
| 1029 | + * A lot of drivers need to generate vblank events for the very next vblank | |
| 1030 | + * interrupt. For example when the page flip interrupt happens when the page | |
| 1031 | + * flip gets armed, but not when it actually executes within the next vblank | |
| 1032 | + * period. This helper function implements exactly the required vblank arming | |
| 1033 | + * behaviour. | |
| 1034 | + * | |
| 1035 | + * Caller must hold event lock. Caller must also hold a vblank reference for | |
| 1036 | + * the event @e, which will be dropped when the next vblank arrives. | |
| 1037 | + * | |
| 1038 | + * This is the native KMS version of drm_arm_vblank_event(). | |
| 1039 | + */ | |
| 1040 | +void drm_crtc_arm_vblank_event(struct drm_crtc *crtc, | |
| 1041 | + struct drm_pending_vblank_event *e) | |
| 1042 | +{ | |
| 1043 | + drm_arm_vblank_event(crtc->dev, drm_crtc_index(crtc), e); | |
| 1044 | +} | |
| 1045 | +EXPORT_SYMBOL(drm_crtc_arm_vblank_event); | |
| 1046 | + | |
| 995 | 1047 | /** |
| 996 | 1048 | * drm_send_vblank_event - helper to send vblank event after pageflip |
| 997 | 1049 | * @dev: DRM device |
| @@ -2734,6 +2734,8 @@ static const char *power_domain_str(enum intel_display_power_domain domain) · 2733 unmodified lines | ||
| 2734 | 2734 | return "AUX_C"; |
| 2735 | 2735 | case POWER_DOMAIN_AUX_D: |
| 2736 | 2736 | return "AUX_D"; |
| 2737 | + case POWER_DOMAIN_GMBUS: | |
| 2738 | + return "GMBUS"; | |
| 2737 | 2739 | case POWER_DOMAIN_INIT: |
| 2738 | 2740 | return "INIT"; |
| 2739 | 2741 | default: |
| @@ -199,6 +199,7 @@ enum intel_display_power_domain { · 198 unmodified lines | ||
| 199 | 199 | POWER_DOMAIN_AUX_B, |
| 200 | 200 | POWER_DOMAIN_AUX_C, |
| 201 | 201 | POWER_DOMAIN_AUX_D, |
| 202 | + POWER_DOMAIN_GMBUS, | |
| 202 | 203 | POWER_DOMAIN_INIT, |
| 203 | 204 | |
| 204 | 205 | POWER_DOMAIN_NUM, |
| @@ -1210,8 +1210,16 @@ int __i915_wait_request(struct drm_i915_gem_request *req, · 1209 unmodified lines | ||
| 1210 | 1210 | if (i915_gem_request_completed(req, true)) |
| 1211 | 1211 | return 0; |
| 1212 | 1212 | |
| 1213 | − timeout_expire = timeout ? | |
| 1214 | − jiffies + nsecs_to_jiffies_timeout((u64)*timeout) : 0; | |
| 1213 | + timeout_expire = 0; | |
| 1214 | + if (timeout) { | |
| 1215 | + if (WARN_ON(*timeout < 0)) | |
| 1216 | + return -EINVAL; | |
| 1217 | + | |
| 1218 | + if (*timeout == 0) | |
| 1219 | + return -ETIME; | |
| 1220 | + | |
| 1221 | + timeout_expire = jiffies + nsecs_to_jiffies_timeout(*timeout); | |
| 1222 | + } | |
| 1215 | 1223 | |
| 1216 | 1224 | if (INTEL_INFO(dev_priv)->gen >= 6) |
| 1217 | 1225 | gen6_rps_boost(dev_priv, rps, req->emitted_jiffies); |
| @@ -642,11 +642,10 @@ i915_gem_detect_bit_6_swizzle(struct drm_device *dev) · 641 unmodified lines | ||
| 642 | 642 | } |
| 643 | 643 | |
| 644 | 644 | /* check for L-shaped memory aka modified enhanced addressing */ |
| 645 | − if (IS_GEN4(dev)) { | |
| 646 | − uint32_t ddc2 = I915_READ(DCC2); | |
| 647 | − | |
| 648 | − if (!(ddc2 & DCC2_MODIFIED_ENHANCED_DISABLE)) | |
| 649 | − dev_priv->quirks |= QUIRK_PIN_SWIZZLED_PAGES; | |
| 645 | + if (IS_GEN4(dev) && | |
| 646 | + !(I915_READ(DCC2) & DCC2_MODIFIED_ENHANCED_DISABLE)) { | |
| 647 | + swizzle_x = I915_BIT_6_SWIZZLE_UNKNOWN; | |
| 648 | + swizzle_y = I915_BIT_6_SWIZZLE_UNKNOWN; | |
| 650 | 649 | } |
| 651 | 650 | |
| 652 | 651 | if (dcc == 0xffffffff) { |
| @@ -675,16 +674,35 @@ i915_gem_detect_bit_6_swizzle(struct drm_device *dev) · 22 unmodified lines | ||
| 675 | 674 | * matching, which was the case for the swizzling required in |
| 676 | 675 | * the table above, or from the 1-ch value being less than |
| 677 | 676 | * the minimum size of a rank. |
| 677 | + * | |
| 678 | + * Reports indicate that the swizzling actually | |
| 679 | + * varies depending upon page placement inside the | |
| 680 | + * channels, i.e. we see swizzled pages where the | |
| 681 | + * banks of memory are paired and unswizzled on the | |
| 682 | + * uneven portion, so leave that as unknown. | |
| 678 | 683 | */ |
| 679 | − if (I915_READ16(C0DRB3) != I915_READ16(C1DRB3)) { | |
| 680 | − swizzle_x = I915_BIT_6_SWIZZLE_NONE; | |
| 681 | − swizzle_y = I915_BIT_6_SWIZZLE_NONE; | |
| 682 | − } else { | |
| 684 | + if (I915_READ16(C0DRB3) == I915_READ16(C1DRB3)) { | |
| 683 | 685 | swizzle_x = I915_BIT_6_SWIZZLE_9_10; |
| 684 | 686 | swizzle_y = I915_BIT_6_SWIZZLE_9; |
| 685 | 687 | } |
| 686 | 688 | } |
| 687 | 689 | |
| 690 | + if (swizzle_x == I915_BIT_6_SWIZZLE_UNKNOWN || | |
| 691 | + swizzle_y == I915_BIT_6_SWIZZLE_UNKNOWN) { | |
| 692 | + /* Userspace likes to explode if it sees unknown swizzling, | |
| 693 | + * so lie. We will finish the lie when reporting through | |
| 694 | + * the get-tiling-ioctl by reporting the physical swizzle | |
| 695 | + * mode as unknown instead. | |
| 696 | + * | |
| 697 | + * As we don't strictly know what the swizzling is, it may be | |
| 698 | + * bit17 dependent, and so we need to also prevent the pages | |
| 699 | + * from being moved. | |
| 700 | + */ | |
| 701 | + dev_priv->quirks |= QUIRK_PIN_SWIZZLED_PAGES; | |
| 702 | + swizzle_x = I915_BIT_6_SWIZZLE_NONE; | |
| 703 | + swizzle_y = I915_BIT_6_SWIZZLE_NONE; | |
| 704 | + } | |
| 705 | + | |
| 688 | 706 | dev_priv->mm.bit_6_swizzle_x = swizzle_x; |
| 689 | 707 | dev_priv->mm.bit_6_swizzle_y = swizzle_y; |
| 690 | 708 | } |
| @@ -5194,11 +5194,31 @@ static enum intel_display_power_domain port_to_power_domain(enum port port) · 5193 unmodified lines | ||
| 5194 | 5194 | case PORT_E: |
| 5195 | 5195 | return POWER_DOMAIN_PORT_DDI_E_2_LANES; |
| 5196 | 5196 | default: |
| 5197 | − WARN_ON_ONCE(1); | |
| 5197 | + MISSING_CASE(port); | |
| 5198 | 5198 | return POWER_DOMAIN_PORT_OTHER; |
| 5199 | 5199 | } |
| 5200 | 5200 | } |
| 5201 | 5201 | |
| 5202 | +static enum intel_display_power_domain port_to_aux_power_domain(enum port port) | |
| 5203 | +{ | |
| 5204 | + switch (port) { | |
| 5205 | + case PORT_A: | |
| 5206 | + return POWER_DOMAIN_AUX_A; | |
| 5207 | + case PORT_B: | |
| 5208 | + return POWER_DOMAIN_AUX_B; | |
| 5209 | + case PORT_C: | |
| 5210 | + return POWER_DOMAIN_AUX_C; | |
| 5211 | + case PORT_D: | |
| 5212 | + return POWER_DOMAIN_AUX_D; | |
| 5213 | + case PORT_E: | |
| 5214 | + /* FIXME: Check VBT for actual wiring of PORT E */ | |
| 5215 | + return POWER_DOMAIN_AUX_D; | |
| 5216 | + default: | |
| 5217 | + MISSING_CASE(port); | |
| 5218 | + return POWER_DOMAIN_AUX_A; | |
| 5219 | + } | |
| 5220 | +} | |
| 5221 | + | |
| 5202 | 5222 | #define for_each_power_domain(domain, mask) \ |
| 5203 | 5223 | for ((domain) = 0; (domain) < POWER_DOMAIN_NUM; (domain)++) \ |
| 5204 | 5224 | if ((1 << (domain)) & (mask)) |
| @@ -5230,6 +5250,36 @@ intel_display_port_power_domain(struct intel_encoder *intel_encoder) · 25 unmodified lines | ||
| 5230 | 5250 | } |
| 5231 | 5251 | } |
| 5232 | 5252 | |
| 5253 | +enum intel_display_power_domain | |
| 5254 | +intel_display_port_aux_power_domain(struct intel_encoder *intel_encoder) | |
| 5255 | +{ | |
| 5256 | + struct drm_device *dev = intel_encoder->base.dev; | |
| 5257 | + struct intel_digital_port *intel_dig_port; | |
| 5258 | + | |
| 5259 | + switch (intel_encoder->type) { | |
| 5260 | + case INTEL_OUTPUT_UNKNOWN: | |
| 5261 | + case INTEL_OUTPUT_HDMI: | |
| 5262 | + /* | |
| 5263 | + * Only DDI platforms should ever use these output types. | |
| 5264 | + * We can get here after the HDMI detect code has already set | |
| 5265 | + * the type of the shared encoder. Since we can't be sure | |
| 5266 | + * what's the status of the given connectors, play safe and | |
| 5267 | + * run the DP detection too. | |
| 5268 | + */ | |
| 5269 | + WARN_ON_ONCE(!HAS_DDI(dev)); | |
| 5270 | + case INTEL_OUTPUT_DISPLAYPORT: | |
| 5271 | + case INTEL_OUTPUT_EDP: | |
| 5272 | + intel_dig_port = enc_to_dig_port(&intel_encoder->base); | |
| 5273 | + return port_to_aux_power_domain(intel_dig_port->port); | |
| 5274 | + case INTEL_OUTPUT_DP_MST: | |
| 5275 | + intel_dig_port = enc_to_mst(&intel_encoder->base)->primary; | |
| 5276 | + return port_to_aux_power_domain(intel_dig_port->port); | |
| 5277 | + default: | |
| 5278 | + MISSING_CASE(intel_encoder->type); | |
| 5279 | + return POWER_DOMAIN_AUX_A; | |
| 5280 | + } | |
| 5281 | +} | |
| 5282 | + | |
| 5233 | 5283 | static unsigned long get_crtc_power_domains(struct drm_crtc *crtc) |
| 5234 | 5284 | { |
| 5235 | 5285 | struct drm_device *dev = crtc->dev; |
| @@ -12460,7 +12510,6 @@ intel_pipe_config_compare(struct drm_device *dev, · 7224 unmodified lines | ||
| 12460 | 12510 | if (INTEL_INFO(dev)->gen < 8) { |
| 12461 | 12511 | PIPE_CONF_CHECK_M_N(dp_m_n); |
| 12462 | 12512 | |
| 12463 | − PIPE_CONF_CHECK_I(has_drrs); | |
| 12464 | 12513 | if (current_config->has_drrs) |
| 12465 | 12514 | PIPE_CONF_CHECK_M_N(dp_m2_n2); |
| 12466 | 12515 | } else |
| @@ -277,7 +277,7 @@ static void pps_lock(struct intel_dp *intel_dp) · 276 unmodified lines | ||
| 277 | 277 | * See vlv_power_sequencer_reset() why we need |
| 278 | 278 | * a power domain reference here. |
| 279 | 279 | */ |
| 280 | − power_domain = intel_display_port_power_domain(encoder); | |
| 280 | + power_domain = intel_display_port_aux_power_domain(encoder); | |
| 281 | 281 | intel_display_power_get(dev_priv, power_domain); |
| 282 | 282 | |
| 283 | 283 | mutex_lock(&dev_priv->pps_mutex); |
| @@ -293,7 +293,7 @@ static void pps_unlock(struct intel_dp *intel_dp) · 9 unmodified lines | ||
| 293 | 293 | |
| 294 | 294 | mutex_unlock(&dev_priv->pps_mutex); |
| 295 | 295 | |
| 296 | − power_domain = intel_display_port_power_domain(encoder); | |
| 296 | + power_domain = intel_display_port_aux_power_domain(encoder); | |
| 297 | 297 | intel_display_power_put(dev_priv, power_domain); |
| 298 | 298 | } |
| 299 | 299 | |
| @@ -816,8 +816,6 @@ intel_dp_aux_ch(struct intel_dp *intel_dp, · 516 unmodified lines | ||
| 816 | 816 | |
| 817 | 817 | intel_dp_check_edp(intel_dp); |
| 818 | 818 | |
| 819 | − intel_aux_display_runtime_get(dev_priv); | |
| 820 | − | |
| 821 | 819 | /* Try to wait for any previous AUX channel activity */ |
| 822 | 820 | for (try = 0; try < 3; try++) { |
| 823 | 821 | status = I915_READ_NOTRACE(ch_ctl); |
| @@ -926,7 +924,6 @@ done: · 102 unmodified lines | ||
| 926 | 924 | ret = recv_bytes; |
| 927 | 925 | out: |
| 928 | 926 | pm_qos_update_request(&dev_priv->pm_qos, PM_QOS_DEFAULT_VALUE); |
| 929 | − intel_aux_display_runtime_put(dev_priv); | |
| 930 | 927 | |
| 931 | 928 | if (vdd) |
| 932 | 929 | edp_panel_vdd_off(intel_dp, false); |
| @@ -1784,7 +1781,7 @@ static bool edp_panel_vdd_on(struct intel_dp *intel_dp) · 851 unmodified lines | ||
| 1784 | 1781 | if (edp_have_panel_vdd(intel_dp)) |
| 1785 | 1782 | return need_to_disable; |
| 1786 | 1783 | |
| 1787 | − power_domain = intel_display_port_power_domain(intel_encoder); | |
| 1784 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 1788 | 1785 | intel_display_power_get(dev_priv, power_domain); |
| 1789 | 1786 | |
| 1790 | 1787 | DRM_DEBUG_KMS("Turning eDP port %c VDD on\n", |
| @@ -1874,7 +1871,7 @@ static void edp_panel_vdd_off_sync(struct intel_dp *intel_dp) · 83 unmodified lines | ||
| 1874 | 1871 | if ((pp & POWER_TARGET_ON) == 0) |
| 1875 | 1872 | intel_dp->last_power_cycle = jiffies; |
| 1876 | 1873 | |
| 1877 | − power_domain = intel_display_port_power_domain(intel_encoder); | |
| 1874 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 1878 | 1875 | intel_display_power_put(dev_priv, power_domain); |
| 1879 | 1876 | } |
| 1880 | 1877 | |
| @@ -2025,7 +2022,7 @@ static void edp_panel_off(struct intel_dp *intel_dp) · 144 unmodified lines | ||
| 2025 | 2022 | wait_panel_off(intel_dp); |
| 2026 | 2023 | |
| 2027 | 2024 | /* We got a reference when we enabled the VDD. */ |
| 2028 | − power_domain = intel_display_port_power_domain(intel_encoder); | |
| 2025 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 2029 | 2026 | intel_display_power_put(dev_priv, power_domain); |
| 2030 | 2027 | } |
| 2031 | 2028 | |
| @@ -4765,26 +4762,6 @@ intel_dp_unset_edid(struct intel_dp *intel_dp) · 2733 unmodified lines | ||
| 4765 | 4762 | intel_dp->has_audio = false; |
| 4766 | 4763 | } |
| 4767 | 4764 | |
| 4768 | −static enum intel_display_power_domain | |
| 4769 | −intel_dp_power_get(struct intel_dp *dp) | |
| 4770 | −{ | |
| 4771 | − struct intel_encoder *encoder = &dp_to_dig_port(dp)->base; | |
| 4772 | − enum intel_display_power_domain power_domain; | |
| 4773 | − | |
| 4774 | − power_domain = intel_display_port_power_domain(encoder); | |
| 4775 | − intel_display_power_get(to_i915(encoder->base.dev), power_domain); | |
| 4776 | − | |
| 4777 | − return power_domain; | |
| 4778 | −} | |
| 4779 | − | |
| 4780 | −static void | |
| 4781 | −intel_dp_power_put(struct intel_dp *dp, | |
| 4782 | − enum intel_display_power_domain power_domain) | |
| 4783 | −{ | |
| 4784 | − struct intel_encoder *encoder = &dp_to_dig_port(dp)->base; | |
| 4785 | − intel_display_power_put(to_i915(encoder->base.dev), power_domain); | |
| 4786 | −} | |
| 4787 | − | |
| 4788 | 4765 | static enum drm_connector_status |
| 4789 | 4766 | intel_dp_detect(struct drm_connector *connector, bool force) |
| 4790 | 4767 | { |
| @@ -4808,7 +4785,8 @@ intel_dp_detect(struct drm_connector *connector, bool force) · 17 unmodified lines | ||
| 4808 | 4785 | return connector_status_disconnected; |
| 4809 | 4786 | } |
| 4810 | 4787 | |
| 4811 | − power_domain = intel_dp_power_get(intel_dp); | |
| 4788 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 4789 | + intel_display_power_get(to_i915(dev), power_domain); | |
| 4812 | 4790 | |
| 4813 | 4791 | /* Can't disconnect eDP, but you can close the lid... */ |
| 4814 | 4792 | if (is_edp(intel_dp)) |
| @@ -4853,7 +4831,7 @@ intel_dp_detect(struct drm_connector *connector, bool force) · 38 unmodified lines | ||
| 4853 | 4831 | } |
| 4854 | 4832 | |
| 4855 | 4833 | out: |
| 4856 | − intel_dp_power_put(intel_dp, power_domain); | |
| 4834 | + intel_display_power_put(to_i915(dev), power_domain); | |
| 4857 | 4835 | return status; |
| 4858 | 4836 | } |
| 4859 | 4837 | |
| @@ -4862,6 +4840,7 @@ intel_dp_force(struct drm_connector *connector) · 2 unmodified lines | ||
| 4862 | 4840 | { |
| 4863 | 4841 | struct intel_dp *intel_dp = intel_attached_dp(connector); |
| 4864 | 4842 | struct intel_encoder *intel_encoder = &dp_to_dig_port(intel_dp)->base; |
| 4843 | + struct drm_i915_private *dev_priv = to_i915(intel_encoder->base.dev); | |
| 4865 | 4844 | enum intel_display_power_domain power_domain; |
| 4866 | 4845 | |
| 4867 | 4846 | DRM_DEBUG_KMS("[CONNECTOR:%d:%s]\n", |
| @@ -4871,11 +4850,12 @@ intel_dp_force(struct drm_connector *connector) · 3 unmodified lines | ||
| 4871 | 4850 | if (connector->status != connector_status_connected) |
| 4872 | 4851 | return; |
| 4873 | 4852 | |
| 4874 | − power_domain = intel_dp_power_get(intel_dp); | |
| 4853 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 4854 | + intel_display_power_get(dev_priv, power_domain); | |
| 4875 | 4855 | |
| 4876 | 4856 | intel_dp_set_edid(intel_dp); |
| 4877 | 4857 | |
| 4878 | − intel_dp_power_put(intel_dp, power_domain); | |
| 4858 | + intel_display_power_put(dev_priv, power_domain); | |
| 4879 | 4859 | |
| 4880 | 4860 | if (intel_encoder->type != INTEL_OUTPUT_EDP) |
| 4881 | 4861 | intel_encoder->type = INTEL_OUTPUT_DISPLAYPORT; |
| @@ -5091,7 +5071,7 @@ static void intel_edp_panel_vdd_sanitize(struct intel_dp *intel_dp) · 209 unmodified lines | ||
| 5091 | 5071 | * indefinitely. |
| 5092 | 5072 | */ |
| 5093 | 5073 | DRM_DEBUG_KMS("VDD left on by BIOS, adjusting state tracking\n"); |
| 5094 | − power_domain = intel_display_port_power_domain(&intel_dig_port->base); | |
| 5074 | + power_domain = intel_display_port_aux_power_domain(&intel_dig_port->base); | |
| 5095 | 5075 | intel_display_power_get(dev_priv, power_domain); |
| 5096 | 5076 | |
| 5097 | 5077 | edp_panel_vdd_schedule_off(intel_dp); |
| @@ -5153,7 +5133,8 @@ intel_dp_hpd_pulse(struct intel_digital_port *intel_dig_port, bool long_hpd) · 55 unmodified lines | ||
| 5153 | 5133 | enum intel_display_power_domain power_domain; |
| 5154 | 5134 | enum irqreturn ret = IRQ_NONE; |
| 5155 | 5135 | |
| 5156 | − if (intel_dig_port->base.type != INTEL_OUTPUT_EDP) | |
| 5136 | + if (intel_dig_port->base.type != INTEL_OUTPUT_EDP && | |
| 5137 | + intel_dig_port->base.type != INTEL_OUTPUT_HDMI) | |
| 5157 | 5138 | intel_dig_port->base.type = INTEL_OUTPUT_DISPLAYPORT; |
| 5158 | 5139 | |
| 5159 | 5140 | if (long_hpd && intel_dig_port->base.type == INTEL_OUTPUT_EDP) { |
| @@ -5172,7 +5153,7 @@ intel_dp_hpd_pulse(struct intel_digital_port *intel_dig_port, bool long_hpd) · 12 unmodified lines | ||
| 5172 | 5153 | port_name(intel_dig_port->port), |
| 5173 | 5154 | long_hpd ? "long" : "short"); |
| 5174 | 5155 | |
| 5175 | − power_domain = intel_display_port_power_domain(intel_encoder); | |
| 5156 | + power_domain = intel_display_port_aux_power_domain(intel_encoder); | |
| 5176 | 5157 | intel_display_power_get(dev_priv, power_domain); |
| 5177 | 5158 | |
| 5178 | 5159 | if (long_hpd) { |
| @@ -1169,6 +1169,8 @@ void hsw_enable_ips(struct intel_crtc *crtc); · 1168 unmodified lines | ||
| 1169 | 1169 | void hsw_disable_ips(struct intel_crtc *crtc); |
| 1170 | 1170 | enum intel_display_power_domain |
| 1171 | 1171 | intel_display_port_power_domain(struct intel_encoder *intel_encoder); |
| 1172 | +enum intel_display_power_domain | |
| 1173 | +intel_display_port_aux_power_domain(struct intel_encoder *intel_encoder); | |
| 1172 | 1174 | void intel_mode_from_pipe_config(struct drm_display_mode *mode, |
| 1173 | 1175 | struct intel_crtc_state *pipe_config); |
| 1174 | 1176 | void intel_crtc_wait_for_pending_flips(struct drm_crtc *crtc); |
| @@ -1377,8 +1379,6 @@ void intel_display_power_get(struct drm_i915_private *dev_priv, · 202 unmodified lines | ||
| 1377 | 1379 | enum intel_display_power_domain domain); |
| 1378 | 1380 | void intel_display_power_put(struct drm_i915_private *dev_priv, |
| 1379 | 1381 | enum intel_display_power_domain domain); |
| 1380 | −void intel_aux_display_runtime_get(struct drm_i915_private *dev_priv); | |
| 1381 | −void intel_aux_display_runtime_put(struct drm_i915_private *dev_priv); | |
| 1382 | 1382 | void intel_runtime_pm_get(struct drm_i915_private *dev_priv); |
| 1383 | 1383 | void intel_runtime_pm_get_noresume(struct drm_i915_private *dev_priv); |
| 1384 | 1384 | void intel_runtime_pm_put(struct drm_i915_private *dev_priv); |
| @@ -1335,21 +1335,17 @@ intel_hdmi_set_edid(struct drm_connector *connector, bool force) · 1334 unmodified lines | ||
| 1335 | 1335 | { |
| 1336 | 1336 | struct drm_i915_private *dev_priv = to_i915(connector->dev); |
| 1337 | 1337 | struct intel_hdmi *intel_hdmi = intel_attached_hdmi(connector); |
| 1338 | − struct intel_encoder *intel_encoder = | |
| 1339 | − &hdmi_to_dig_port(intel_hdmi)->base; | |
| 1340 | − enum intel_display_power_domain power_domain; | |
| 1341 | 1338 | struct edid *edid = NULL; |
| 1342 | 1339 | bool connected = false; |
| 1343 | 1340 | |
| 1344 | − power_domain = intel_display_port_power_domain(intel_encoder); | |
| 1345 | − intel_display_power_get(dev_priv, power_domain); | |
| 1341 | + intel_display_power_get(dev_priv, POWER_DOMAIN_GMBUS); | |
| 1346 | 1342 | |
| 1347 | 1343 | if (force) |
| 1348 | 1344 | edid = drm_get_edid(connector, |
| 1349 | 1345 | intel_gmbus_get_adapter(dev_priv, |
| 1350 | 1346 | intel_hdmi->ddc_bus)); |
| 1351 | 1347 | |
| 1352 | − intel_display_power_put(dev_priv, power_domain); | |
| 1348 | + intel_display_power_put(dev_priv, POWER_DOMAIN_GMBUS); | |
| 1353 | 1349 | |
| 1354 | 1350 | to_intel_connector(connector)->detect_edid = edid; |
| 1355 | 1351 | if (edid && edid->input & DRM_EDID_INPUT_DIGITAL) { |
| @@ -1383,6 +1379,8 @@ intel_hdmi_detect(struct drm_connector *connector, bool force) · 27 unmodified lines | ||
| 1383 | 1379 | DRM_DEBUG_KMS("[CONNECTOR:%d:%s]\n", |
| 1384 | 1380 | connector->base.id, connector->name); |
| 1385 | 1381 | |
| 1382 | + intel_display_power_get(dev_priv, POWER_DOMAIN_GMBUS); | |
| 1383 | + | |
| 1386 | 1384 | while (!live_status && --retry) { |
| 1387 | 1385 | live_status = intel_digital_port_connected(dev_priv, |
| 1388 | 1386 | hdmi_to_dig_port(intel_hdmi)); |
| @@ -1402,6 +1400,8 @@ intel_hdmi_detect(struct drm_connector *connector, bool force) · 13 unmodified lines | ||
| 1402 | 1400 | } else |
| 1403 | 1401 | status = connector_status_disconnected; |
| 1404 | 1402 | |
| 1403 | + intel_display_power_put(dev_priv, POWER_DOMAIN_GMBUS); | |
| 1404 | + | |
| 1405 | 1405 | return status; |
| 1406 | 1406 | } |
| 1407 | 1407 | |
| @@ -483,7 +483,7 @@ gmbus_xfer(struct i2c_adapter *adapter, · 482 unmodified lines | ||
| 483 | 483 | int i = 0, inc, try = 0; |
| 484 | 484 | int ret = 0; |
| 485 | 485 | |
| 486 | − intel_aux_display_runtime_get(dev_priv); | |
| 486 | + intel_display_power_get(dev_priv, POWER_DOMAIN_GMBUS); | |
| 487 | 487 | mutex_lock(&dev_priv->gmbus_mutex); |
| 488 | 488 | |
| 489 | 489 | if (bus->force_bit) { |
| @@ -595,7 +595,9 @@ timeout: · 105 unmodified lines | ||
| 595 | 595 | |
| 596 | 596 | out: |
| 597 | 597 | mutex_unlock(&dev_priv->gmbus_mutex); |
| 598 | − intel_aux_display_runtime_put(dev_priv); | |
| 598 | + | |
| 599 | + intel_display_power_put(dev_priv, POWER_DOMAIN_GMBUS); | |
| 600 | + | |
| 599 | 601 | return ret; |
| 600 | 602 | } |
| 601 | 603 | |
| @@ -362,6 +362,7 @@ static void hsw_set_power_well(struct drm_i915_private *dev_priv, · 361 unmodified lines | ||
| 362 | 362 | BIT(POWER_DOMAIN_AUX_C) | \ |
| 363 | 363 | BIT(POWER_DOMAIN_AUDIO) | \ |
| 364 | 364 | BIT(POWER_DOMAIN_VGA) | \ |
| 365 | + BIT(POWER_DOMAIN_GMBUS) | \ | |
| 365 | 366 | BIT(POWER_DOMAIN_INIT)) |
| 366 | 367 | #define BXT_DISPLAY_POWERWELL_1_POWER_DOMAINS ( \ |
| 367 | 368 | BXT_DISPLAY_POWERWELL_2_POWER_DOMAINS | \ |
| @@ -1483,6 +1484,7 @@ void intel_display_power_put(struct drm_i915_private *dev_priv, · 1115 unmodified lines | ||
| 1483 | 1484 | BIT(POWER_DOMAIN_AUX_B) | \ |
| 1484 | 1485 | BIT(POWER_DOMAIN_AUX_C) | \ |
| 1485 | 1486 | BIT(POWER_DOMAIN_AUX_D) | \ |
| 1487 | + BIT(POWER_DOMAIN_GMBUS) | \ | |
| 1486 | 1488 | BIT(POWER_DOMAIN_INIT)) |
| 1487 | 1489 | #define HSW_DISPLAY_POWER_DOMAINS ( \ |
| 1488 | 1490 | (POWER_DOMAIN_MASK & ~HSW_ALWAYS_ON_POWER_DOMAINS) | \ |
| @@ -1845,6 +1847,8 @@ int intel_power_domains_init(struct drm_i915_private *dev_priv) · 356 unmodified lines | ||
| 1845 | 1847 | i915.disable_power_well = sanitize_disable_power_well_option(dev_priv, |
| 1846 | 1848 | i915.disable_power_well); |
| 1847 | 1849 | |
| 1850 | + BUILD_BUG_ON(POWER_DOMAIN_NUM > 31); | |
| 1851 | + | |
| 1848 | 1852 | mutex_init(&power_domains->lock); |
| 1849 | 1853 | |
| 1850 | 1854 | /* |
| @@ -2063,36 +2067,6 @@ void intel_power_domains_init_hw(struct drm_i915_private *dev_priv) · 212 unmodified lines | ||
| 2063 | 2067 | power_domains->initializing = false; |
| 2064 | 2068 | } |
| 2065 | 2069 | |
| 2066 | −/** | |
| 2067 | − * intel_aux_display_runtime_get - grab an auxiliary power domain reference | |
| 2068 | − * @dev_priv: i915 device instance | |
| 2069 | − * | |
| 2070 | − * This function grabs a power domain reference for the auxiliary power domain | |
| 2071 | − * (for access to the GMBUS and DP AUX blocks) and ensures that it and all its | |
| 2072 | − * parents are powered up. Therefore users should only grab a reference to the | |
| 2073 | − * innermost power domain they need. | |
| 2074 | − * | |
| 2075 | − * Any power domain reference obtained by this function must have a symmetric | |
| 2076 | − * call to intel_aux_display_runtime_put() to release the reference again. | |
| 2077 | − */ | |
| 2078 | −void intel_aux_display_runtime_get(struct drm_i915_private *dev_priv) | |
| 2079 | −{ | |
| 2080 | − intel_runtime_pm_get(dev_priv); | |
| 2081 | −} | |
| 2082 | − | |
| 2083 | −/** | |
| 2084 | − * intel_aux_display_runtime_put - release an auxiliary power domain reference | |
| 2085 | − * @dev_priv: i915 device instance | |
| 2086 | − * | |
| 2087 | − * This function drops the auxiliary power domain reference obtained by | |
| 2088 | − * intel_aux_display_runtime_get() and might power down the corresponding | |
| 2089 | − * hardware block right away if this is the last reference. | |
| 2090 | − */ | |
| 2091 | −void intel_aux_display_runtime_put(struct drm_i915_private *dev_priv) | |
| 2092 | −{ | |
| 2093 | − intel_runtime_pm_put(dev_priv); | |
| 2094 | −} | |
| 2095 | − | |
| 2096 | 2070 | /** |
| 2097 | 2071 | * intel_runtime_pm_get - grab a runtime pm reference |
| 2098 | 2072 | * @dev_priv: i915 device instance |
| @@ -63,8 +63,7 @@ static void imx_drm_driver_lastclose(struct drm_device *drm) · 62 unmodified lines | ||
| 63 | 63 | #if IS_ENABLED(CONFIG_DRM_IMX_FB_HELPER) |
| 64 | 64 | struct imx_drm_device *imxdrm = drm->dev_private; |
| 65 | 65 | |
| 66 | − if (imxdrm->fbhelper) | |
| 67 | − drm_fbdev_cma_restore_mode(imxdrm->fbhelper); | |
| 66 | + drm_fbdev_cma_restore_mode(imxdrm->fbhelper); | |
| 68 | 67 | #endif |
| 69 | 68 | } |
| 70 | 69 | |
| @@ -340,7 +339,7 @@ err_kms: · 269 unmodified lines | ||
| 340 | 339 | * imx_drm_add_crtc - add a new crtc |
| 341 | 340 | */ |
| 342 | 341 | int imx_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc, |
| 343 | − struct imx_drm_crtc **new_crtc, | |
| 342 | + struct imx_drm_crtc **new_crtc, struct drm_plane *primary_plane, | |
| 344 | 343 | const struct imx_drm_crtc_helper_funcs *imx_drm_helper_funcs, |
| 345 | 344 | struct device_node *port) |
| 346 | 345 | { |
| @@ -379,7 +378,7 @@ int imx_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc, · 32 unmodified lines | ||
| 379 | 378 | drm_crtc_helper_add(crtc, |
| 380 | 379 | imx_drm_crtc->imx_drm_helper_funcs.crtc_helper_funcs); |
| 381 | 380 | |
| 382 | − drm_crtc_init(drm, crtc, | |
| 381 | + drm_crtc_init_with_planes(drm, crtc, primary_plane, NULL, | |
| 383 | 382 | imx_drm_crtc->imx_drm_helper_funcs.crtc_funcs); |
| 384 | 383 | |
| 385 | 384 | return 0; |
| @@ -9,6 +9,7 @@ struct drm_display_mode; · 8 unmodified lines | ||
| 9 | 9 | struct drm_encoder; |
| 10 | 10 | struct drm_fbdev_cma; |
| 11 | 11 | struct drm_framebuffer; |
| 12 | +struct drm_plane; | |
| 12 | 13 | struct imx_drm_crtc; |
| 13 | 14 | struct platform_device; |
| 14 | 15 | |
| @@ -24,7 +25,7 @@ struct imx_drm_crtc_helper_funcs { · 9 unmodified lines | ||
| 24 | 25 | }; |
| 25 | 26 | |
| 26 | 27 | int imx_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc, |
| 27 | − struct imx_drm_crtc **new_crtc, | |
| 28 | + struct imx_drm_crtc **new_crtc, struct drm_plane *primary_plane, | |
| 28 | 29 | const struct imx_drm_crtc_helper_funcs *imx_helper_funcs, |
| 29 | 30 | struct device_node *port); |
| 30 | 31 | int imx_drm_remove_crtc(struct imx_drm_crtc *); |
| @@ -721,6 +721,7 @@ static const struct of_device_id imx_tve_dt_ids[] = { · 720 unmodified lines | ||
| 721 | 721 | { .compatible = "fsl,imx53-tve", }, |
| 722 | 722 | { /* sentinel */ } |
| 723 | 723 | }; |
| 724 | +MODULE_DEVICE_TABLE(of, imx_tve_dt_ids); | |
| 724 | 725 | |
| 725 | 726 | static struct platform_driver imx_tve_driver = { |
| 726 | 727 | .probe = imx_tve_probe, |
| @@ -212,7 +212,8 @@ static void ipu_crtc_handle_pageflip(struct ipu_crtc *ipu_crtc) · 211 unmodified lines | ||
| 212 | 212 | |
| 213 | 213 | spin_lock_irqsave(&drm->event_lock, flags); |
| 214 | 214 | if (ipu_crtc->page_flip_event) |
| 215 | − drm_send_vblank_event(drm, -1, ipu_crtc->page_flip_event); | |
| 215 | + drm_crtc_send_vblank_event(&ipu_crtc->base, | |
| 216 | + ipu_crtc->page_flip_event); | |
| 216 | 217 | ipu_crtc->page_flip_event = NULL; |
| 217 | 218 | imx_drm_crtc_vblank_put(ipu_crtc->imx_crtc); |
| 218 | 219 | spin_unlock_irqrestore(&drm->event_lock, flags); |
| @@ -349,7 +350,6 @@ static int ipu_crtc_init(struct ipu_crtc *ipu_crtc, · 130 unmodified lines | ||
| 349 | 350 | struct ipu_soc *ipu = dev_get_drvdata(ipu_crtc->dev->parent); |
| 350 | 351 | int dp = -EINVAL; |
| 351 | 352 | int ret; |
| 352 | − int id; | |
| 353 | 353 | |
| 354 | 354 | ret = ipu_get_resources(ipu_crtc, pdata); |
| 355 | 355 | if (ret) { |
| @@ -358,18 +358,23 @@ static int ipu_crtc_init(struct ipu_crtc *ipu_crtc, · 2 unmodified lines | ||
| 358 | 358 | return ret; |
| 359 | 359 | } |
| 360 | 360 | |
| 361 | + if (pdata->dp >= 0) | |
| 362 | + dp = IPU_DP_FLOW_SYNC_BG; | |
| 363 | + ipu_crtc->plane[0] = ipu_plane_init(drm, ipu, pdata->dma[0], dp, 0, | |
| 364 | + DRM_PLANE_TYPE_PRIMARY); | |
| 365 | + if (IS_ERR(ipu_crtc->plane[0])) { | |
| 366 | + ret = PTR_ERR(ipu_crtc->plane[0]); | |
| 367 | + goto err_put_resources; | |
| 368 | + } | |
| 369 | + | |
| 361 | 370 | ret = imx_drm_add_crtc(drm, &ipu_crtc->base, &ipu_crtc->imx_crtc, |
| 362 | − &ipu_crtc_helper_funcs, ipu_crtc->dev->of_node); | |
| 371 | + &ipu_crtc->plane[0]->base, &ipu_crtc_helper_funcs, | |
| 372 | + ipu_crtc->dev->of_node); | |
| 363 | 373 | if (ret) { |
| 364 | 374 | dev_err(ipu_crtc->dev, "adding crtc failed with %d.\n", ret); |
| 365 | 375 | goto err_put_resources; |
| 366 | 376 | } |
| 367 | 377 | |
| 368 | − if (pdata->dp >= 0) | |
| 369 | − dp = IPU_DP_FLOW_SYNC_BG; | |
| 370 | − id = imx_drm_crtc_id(ipu_crtc->imx_crtc); | |
| 371 | − ipu_crtc->plane[0] = ipu_plane_init(ipu_crtc->base.dev, ipu, | |
| 372 | − pdata->dma[0], dp, BIT(id), true); | |
| 373 | 378 | ret = ipu_plane_get_resources(ipu_crtc->plane[0]); |
| 374 | 379 | if (ret) { |
| 375 | 380 | dev_err(ipu_crtc->dev, "getting plane 0 resources failed with %d.\n", |
| @@ -379,10 +384,10 @@ static int ipu_crtc_init(struct ipu_crtc *ipu_crtc, · 3 unmodified lines | ||
| 379 | 384 | |
| 380 | 385 | /* If this crtc is using the DP, add an overlay plane */ |
| 381 | 386 | if (pdata->dp >= 0 && pdata->dma[1] > 0) { |
| 382 | − ipu_crtc->plane[1] = ipu_plane_init(ipu_crtc->base.dev, ipu, | |
| 383 | − pdata->dma[1], | |
| 384 | − IPU_DP_FLOW_SYNC_FG, | |
| 385 | − BIT(id), false); | |
| 387 | + ipu_crtc->plane[1] = ipu_plane_init(drm, ipu, pdata->dma[1], | |
| 388 | + IPU_DP_FLOW_SYNC_FG, | |
| 389 | + drm_crtc_mask(&ipu_crtc->base), | |
| 390 | + DRM_PLANE_TYPE_OVERLAY); | |
| 386 | 391 | if (IS_ERR(ipu_crtc->plane[1])) |
| 387 | 392 | ipu_crtc->plane[1] = NULL; |
| 388 | 393 | } |
| @@ -407,28 +412,6 @@ err_put_resources: · 18 unmodified lines | ||
| 407 | 412 | return ret; |
| 408 | 413 | } |
| 409 | 414 | |
| 410 | −static struct device_node *ipu_drm_get_port_by_id(struct device_node *parent, | |
| 411 | − int port_id) | |
| 412 | −{ | |
| 413 | − struct device_node *port; | |
| 414 | − int id, ret; | |
| 415 | − | |
| 416 | − port = of_get_child_by_name(parent, "port"); | |
| 417 | − while (port) { | |
| 418 | − ret = of_property_read_u32(port, "reg", &id); | |
| 419 | − if (!ret && id == port_id) | |
| 420 | − return port; | |
| 421 | − | |
| 422 | − do { | |
| 423 | − port = of_get_next_child(parent, port); | |
| 424 | − if (!port) | |
| 425 | − return NULL; | |
| 426 | − } while (of_node_cmp(port->name, "port")); | |
| 427 | − } | |
| 428 | − | |
| 429 | − return NULL; | |
| 430 | −} | |
| 431 | − | |
| 432 | 415 | static int ipu_drm_bind(struct device *dev, struct device *master, void *data) |
| 433 | 416 | { |
| 434 | 417 | struct ipu_client_platformdata *pdata = dev->platform_data; |
| @@ -470,23 +453,11 @@ static const struct component_ops ipu_crtc_ops = { · 35 unmodified lines | ||
| 470 | 453 | static int ipu_drm_probe(struct platform_device *pdev) |
| 471 | 454 | { |
| 472 | 455 | struct device *dev = &pdev->dev; |
| 473 | − struct ipu_client_platformdata *pdata = dev->platform_data; | |
| 474 | 456 | int ret; |
| 475 | 457 | |
| 476 | 458 | if (!dev->platform_data) |
| 477 | 459 | return -EINVAL; |
| 478 | 460 | |
| 479 | − if (!dev->of_node) { | |
| 480 | − /* Associate crtc device with the corresponding DI port node */ | |
| 481 | − dev->of_node = ipu_drm_get_port_by_id(dev->parent->of_node, | |
| 482 | − pdata->di + 2); | |
| 483 | − if (!dev->of_node) { | |
| 484 | − dev_err(dev, "missing port@%d node in %s\n", | |
| 485 | − pdata->di + 2, dev->parent->of_node->full_name); | |
| 486 | − return -ENODEV; | |
| 487 | − } | |
| 488 | − } | |
| 489 | − | |
| 490 | 461 | ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)); |
| 491 | 462 | if (ret) |
| 492 | 463 | return ret; |
| @@ -381,7 +381,7 @@ static struct drm_plane_funcs ipu_plane_funcs = { · 380 unmodified lines | ||
| 381 | 381 | |
| 382 | 382 | struct ipu_plane *ipu_plane_init(struct drm_device *dev, struct ipu_soc *ipu, |
| 383 | 383 | int dma, int dp, unsigned int possible_crtcs, |
| 384 | − bool priv) | |
| 384 | + enum drm_plane_type type) | |
| 385 | 385 | { |
| 386 | 386 | struct ipu_plane *ipu_plane; |
| 387 | 387 | int ret; |
| @@ -399,10 +399,9 @@ struct ipu_plane *ipu_plane_init(struct drm_device *dev, struct ipu_soc *ipu, · 11 unmodified lines | ||
| 399 | 399 | ipu_plane->dma = dma; |
| 400 | 400 | ipu_plane->dp_flow = dp; |
| 401 | 401 | |
| 402 | − ret = drm_plane_init(dev, &ipu_plane->base, possible_crtcs, | |
| 403 | − &ipu_plane_funcs, ipu_plane_formats, | |
| 404 | − ARRAY_SIZE(ipu_plane_formats), | |
| 405 | − priv); | |
| 402 | + ret = drm_universal_plane_init(dev, &ipu_plane->base, possible_crtcs, | |
| 403 | + &ipu_plane_funcs, ipu_plane_formats, | |
| 404 | + ARRAY_SIZE(ipu_plane_formats), type); | |
| 406 | 405 | if (ret) { |
| 407 | 406 | DRM_ERROR("failed to initialize plane\n"); |
| 408 | 407 | kfree(ipu_plane); |
| @@ -34,7 +34,7 @@ struct ipu_plane { · 33 unmodified lines | ||
| 34 | 34 | |
| 35 | 35 | struct ipu_plane *ipu_plane_init(struct drm_device *dev, struct ipu_soc *ipu, |
| 36 | 36 | int dma, int dp, unsigned int possible_crtcs, |
| 37 | − bool priv); | |
| 37 | + enum drm_plane_type type); | |
| 38 | 38 | |
| 39 | 39 | /* Init IDMAC, DMFC, DP */ |
| 40 | 40 | int ipu_plane_mode_set(struct ipu_plane *plane, struct drm_crtc *crtc, |
| @@ -54,7 +54,11 @@ static int imx_pd_connector_get_modes(struct drm_connector *connector) · 53 unmodified lines | ||
| 54 | 54 | |
| 55 | 55 | if (imxpd->panel && imxpd->panel->funcs && |
| 56 | 56 | imxpd->panel->funcs->get_modes) { |
| 57 | + struct drm_display_info *di = &connector->display_info; | |
| 58 | + | |
| 57 | 59 | num_modes = imxpd->panel->funcs->get_modes(imxpd->panel); |
| 60 | + if (!imxpd->bus_format && di->num_bus_formats) | |
| 61 | + imxpd->bus_format = di->bus_formats[0]; | |
| 58 | 62 | if (num_modes > 0) |
| 59 | 63 | return num_modes; |
| 60 | 64 | } |
| @@ -159,7 +159,6 @@ struct nvkm_device_func { · 158 unmodified lines | ||
| 159 | 159 | struct nvkm_device_quirk { |
| 160 | 160 | u8 tv_pin_mask; |
| 161 | 161 | u8 tv_gpio; |
| 162 | − bool War00C800_0; | |
| 163 | 162 | }; |
| 164 | 163 | |
| 165 | 164 | struct nvkm_device_chip { |
| @@ -7,6 +7,7 @@ struct nvkm_instmem { · 6 unmodified lines | ||
| 7 | 7 | const struct nvkm_instmem_func *func; |
| 8 | 8 | struct nvkm_subdev subdev; |
| 9 | 9 | |
| 10 | + spinlock_t lock; | |
| 10 | 11 | struct list_head list; |
| 11 | 12 | u32 reserved; |
| 12 | 13 | |
| @@ -367,6 +367,7 @@ static int nouveau_rom_call(acpi_handle rom_handle, uint8_t *bios, · 366 unmodified lines | ||
| 367 | 367 | return -ENODEV; |
| 368 | 368 | } |
| 369 | 369 | obj = (union acpi_object *)buffer.pointer; |
| 370 | + len = min(len, (int)obj->buffer.length); | |
| 370 | 371 | memcpy(bios+offset, obj->buffer.pointer, len); |
| 371 | 372 | kfree(buffer.pointer); |
| 372 | 373 | return len; |
| @@ -829,7 +829,6 @@ nouveau_finish_page_flip(struct nouveau_channel *chan, · 828 unmodified lines | ||
| 829 | 829 | struct drm_device *dev = drm->dev; |
| 830 | 830 | struct nouveau_page_flip_state *s; |
| 831 | 831 | unsigned long flags; |
| 832 | − int crtcid = -1; | |
| 833 | 832 | |
| 834 | 833 | spin_lock_irqsave(&dev->event_lock, flags); |
| 835 | 834 | |
| @@ -841,15 +840,19 @@ nouveau_finish_page_flip(struct nouveau_channel *chan, · 5 unmodified lines | ||
| 841 | 840 | |
| 842 | 841 | s = list_first_entry(&fctx->flip, struct nouveau_page_flip_state, head); |
| 843 | 842 | if (s->event) { |
| 844 | − /* Vblank timestamps/counts are only correct on >= NV-50 */ | |
| 845 | − if (drm->device.info.family >= NV_DEVICE_INFO_V0_TESLA) | |
| 846 | − crtcid = s->crtc; | |
| 843 | + if (drm->device.info.family < NV_DEVICE_INFO_V0_TESLA) { | |
| 844 | + drm_arm_vblank_event(dev, s->crtc, s->event); | |
| 845 | + } else { | |
| 846 | + drm_send_vblank_event(dev, s->crtc, s->event); | |
| 847 | 847 | |
| 848 | − drm_send_vblank_event(dev, crtcid, s->event); | |
| 848 | + /* Give up ownership of vblank for page-flipped crtc */ | |
| 849 | + drm_vblank_put(dev, s->crtc); | |
| 850 | + } | |
| 851 | + } | |
| 852 | + else { | |
| 853 | + /* Give up ownership of vblank for page-flipped crtc */ | |
| 854 | + drm_vblank_put(dev, s->crtc); | |
| 849 | 855 | } |
| 850 | − | |
| 851 | − /* Give up ownership of vblank for page-flipped crtc */ | |
| 852 | − drm_vblank_put(dev, s->crtc); | |
| 853 | 856 | |
| 854 | 857 | list_del(&s->head); |
| 855 | 858 | if (ps) |
| @@ -39,6 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | |
| 40 | 40 | #include <nvif/client.h> |
| 41 | 41 | #include <nvif/device.h> |
| 42 | +#include <nvif/ioctl.h> | |
| 42 | 43 | |
| 43 | 44 | #include <drmP.h> |
| 44 | 45 | |
| @@ -65,9 +66,10 @@ struct nouveau_drm_tile { · 20 unmodified lines | ||
| 65 | 66 | }; |
| 66 | 67 | |
| 67 | 68 | enum nouveau_drm_object_route { |
| 68 | − NVDRM_OBJECT_NVIF = 0, | |
| 69 | + NVDRM_OBJECT_NVIF = NVIF_IOCTL_V0_OWNER_NVIF, | |
| 69 | 70 | NVDRM_OBJECT_USIF, |
| 70 | 71 | NVDRM_OBJECT_ABI16, |
| 72 | + NVDRM_OBJECT_ANY = NVIF_IOCTL_V0_OWNER_ANY, | |
| 71 | 73 | }; |
| 72 | 74 | |
| 73 | 75 | enum nouveau_drm_notify_route { |
| @@ -313,7 +313,10 @@ usif_ioctl(struct drm_file *filp, void __user *user, u32 argc) · 312 unmodified lines | ||
| 313 | 313 | if (nvif_unpack(argv->v0, 0, 0, true)) { |
| 314 | 314 | /* block access to objects not created via this interface */ |
| 315 | 315 | owner = argv->v0.owner; |
| 316 | − argv->v0.owner = NVDRM_OBJECT_USIF; | |
| 316 | + if (argv->v0.object == 0ULL) | |
| 317 | + argv->v0.owner = NVDRM_OBJECT_ANY; /* except client */ | |
| 318 | + else | |
| 319 | + argv->v0.owner = NVDRM_OBJECT_USIF; | |
| 317 | 320 | } else |
| 318 | 321 | goto done; |
| 319 | 322 | |
| @@ -258,12 +258,6 @@ nvkm_device_pci_10de_0df4[] = { · 257 unmodified lines | ||
| 258 | 258 | {} |
| 259 | 259 | }; |
| 260 | 260 | |
| 261 | −static const struct nvkm_device_pci_vendor | |
| 262 | −nvkm_device_pci_10de_0fcd[] = { | |
| 263 | − { 0x17aa, 0x3801, NULL, { .War00C800_0 = true } }, /* Lenovo Y510P */ | |
| 264 | − {} | |
| 265 | −}; | |
| 266 | − | |
| 267 | 261 | static const struct nvkm_device_pci_vendor |
| 268 | 262 | nvkm_device_pci_10de_0fd2[] = { |
| 269 | 263 | { 0x1028, 0x0595, "GeForce GT 640M LE" }, |
| @@ -684,7 +678,6 @@ nvkm_device_pci_10de_1189[] = { · 414 unmodified lines | ||
| 684 | 678 | static const struct nvkm_device_pci_vendor |
| 685 | 679 | nvkm_device_pci_10de_1199[] = { |
| 686 | 680 | { 0x1458, 0xd001, "GeForce GTX 760" }, |
| 687 | − { 0x1462, 0x1106, "GeForce GTX 780M", { .War00C800_0 = true } }, /* Medion Erazer X7827 */ | |
| 688 | 681 | {} |
| 689 | 682 | }; |
| 690 | 683 | |
| @@ -694,14 +687,6 @@ nvkm_device_pci_10de_11e3[] = { · 3 unmodified lines | ||
| 694 | 687 | {} |
| 695 | 688 | }; |
| 696 | 689 | |
| 697 | −static const struct nvkm_device_pci_vendor | |
| 698 | −nvkm_device_pci_10de_11fc[] = { | |
| 699 | − { 0x1179, 0x0001, NULL, { .War00C800_0 = true } }, /* Toshiba Tecra W50 */ | |
| 700 | − { 0x17aa, 0x2211, NULL, { .War00C800_0 = true } }, /* Lenovo W541 */ | |
| 701 | − { 0x17aa, 0x221e, NULL, { .War00C800_0 = true } }, /* Lenovo W541 */ | |
| 702 | − {} | |
| 703 | −}; | |
| 704 | − | |
| 705 | 690 | static const struct nvkm_device_pci_vendor |
| 706 | 691 | nvkm_device_pci_10de_1247[] = { |
| 707 | 692 | { 0x1043, 0x212a, "GeForce GT 635M" }, |
| @@ -1356,7 +1341,7 @@ nvkm_device_pci_10de[] = { · 648 unmodified lines | ||
| 1356 | 1341 | { 0x0fc6, "GeForce GTX 650" }, |
| 1357 | 1342 | { 0x0fc8, "GeForce GT 740" }, |
| 1358 | 1343 | { 0x0fc9, "GeForce GT 730" }, |
| 1359 | − { 0x0fcd, "GeForce GT 755M", nvkm_device_pci_10de_0fcd }, | |
| 1344 | + { 0x0fcd, "GeForce GT 755M" }, | |
| 1360 | 1345 | { 0x0fce, "GeForce GT 640M LE" }, |
| 1361 | 1346 | { 0x0fd1, "GeForce GT 650M" }, |
| 1362 | 1347 | { 0x0fd2, "GeForce GT 640M", nvkm_device_pci_10de_0fd2 }, |
| @@ -1490,7 +1475,7 @@ nvkm_device_pci_10de[] = { · 127 unmodified lines | ||
| 1490 | 1475 | { 0x11e2, "GeForce GTX 765M" }, |
| 1491 | 1476 | { 0x11e3, "GeForce GTX 760M", nvkm_device_pci_10de_11e3 }, |
| 1492 | 1477 | { 0x11fa, "Quadro K4000" }, |
| 1493 | − { 0x11fc, "Quadro K2100M", nvkm_device_pci_10de_11fc }, | |
| 1478 | + { 0x11fc, "Quadro K2100M" }, | |
| 1494 | 1479 | { 0x1200, "GeForce GTX 560 Ti" }, |
| 1495 | 1480 | { 0x1201, "GeForce GTX 560" }, |
| 1496 | 1481 | { 0x1203, "GeForce GTX 460 SE v2" }, |
| @@ -207,6 +207,8 @@ gf117_grctx_generate_attrib(struct gf100_grctx *info) · 206 unmodified lines | ||
| 207 | 207 | const u32 b = beta * gr->ppc_tpc_nr[gpc][ppc]; |
| 208 | 208 | const u32 t = timeslice_mode; |
| 209 | 209 | const u32 o = PPC_UNIT(gpc, ppc, 0); |
| 210 | + if (!(gr->ppc_mask[gpc] & (1 << ppc))) | |
| 211 | + continue; | |
| 210 | 212 | mmio_skip(info, o + 0xc0, (t << 28) | (b << 16) | ++bo); |
| 211 | 213 | mmio_wr32(info, o + 0xc0, (t << 28) | (b << 16) | --bo); |
| 212 | 214 | bo += grctx->attrib_nr_max * gr->ppc_tpc_nr[gpc][ppc]; |
| @@ -52,10 +52,12 @@ mmio_list_base: · 51 unmodified lines | ||
| 52 | 52 | #endif |
| 53 | 53 | |
| 54 | 54 | #ifdef INCLUDE_CODE |
| 55 | +#define gpc_addr(reg,addr) /* | |
| 56 | +*/ imm32(reg,addr) /* | |
| 57 | +*/ or reg NV_PGRAPH_GPCX_GPCCS_MMIO_CTRL_BASE_ENABLE | |
| 55 | 58 | #define gpc_wr32(addr,reg) /* |
| 59 | +*/ gpc_addr($r14,addr) /* | |
| 56 | 60 | */ mov b32 $r15 reg /* |
| 57 | −*/ imm32($r14, addr) /* | |
| 58 | −*/ or $r14 NV_PGRAPH_GPCX_GPCCS_MMIO_CTRL_BASE_ENABLE /* | |
| 59 | 61 | */ call(nv_wr32) |
| 60 | 62 | |
| 61 | 63 | // reports an exception to the host |
| @@ -161,7 +163,7 @@ init: · 99 unmodified lines | ||
| 161 | 163 | |
| 162 | 164 | #if NV_PGRAPH_GPCX_UNK__SIZE > 0 |
| 163 | 165 | // figure out which, and how many, UNKs are actually present |
| 164 | − imm32($r14, 0x500c30) | |
| 166 | + gpc_addr($r14, 0x500c30) | |
| 165 | 167 | clear b32 $r2 |
| 166 | 168 | clear b32 $r3 |
| 167 | 169 | clear b32 $r4 |
| @@ -314,7 +314,7 @@ uint32_t gf117_grgpc_code[] = { · 313 unmodified lines | ||
| 314 | 314 | 0x03f01200, |
| 315 | 315 | 0x0002d000, |
| 316 | 316 | 0x17f104bd, |
| 317 | − 0x10fe0542, | |
| 317 | + 0x10fe0545, | |
| 318 | 318 | 0x0007f100, |
| 319 | 319 | 0x0003f007, |
| 320 | 320 | 0xbd0000d0, |
| @@ -338,184 +338,184 @@ uint32_t gf117_grgpc_code[] = { · 17 unmodified lines | ||
| 338 | 338 | 0x02d00103, |
| 339 | 339 | 0xf104bd00, |
| 340 | 340 | 0xf00c30e7, |
| 341 | − 0x24bd50e3, | |
| 342 | − 0x44bd34bd, | |
| 343 | −/* 0x0430: init_unk_loop */ | |
| 344 | − 0xb06821f4, | |
| 345 | − 0x0bf400f6, | |
| 346 | − 0x01f7f00f, | |
| 347 | − 0xfd04f2bb, | |
| 348 | − 0x30b6054f, | |
| 349 | −/* 0x0445: init_unk_next */ | |
| 350 | − 0x0120b601, | |
| 351 | − 0xb004e0b6, | |
| 352 | − 0x1bf40126, | |
| 353 | −/* 0x0451: init_unk_done */ | |
| 354 | − 0x070380e2, | |
| 355 | − 0xf1080480, | |
| 356 | − 0xf0010027, | |
| 357 | − 0x22cf0223, | |
| 358 | − 0x9534bd00, | |
| 359 | − 0x07f10825, | |
| 360 | − 0x03f0c000, | |
| 361 | − 0x0005d001, | |
| 362 | − 0x07f104bd, | |
| 363 | − 0x03f0c100, | |
| 364 | − 0x0005d001, | |
| 365 | − 0x0e9804bd, | |
| 366 | − 0x010f9800, | |
| 367 | − 0x015021f5, | |
| 368 | − 0xbb002fbb, | |
| 369 | − 0x0e98003f, | |
| 370 | − 0x020f9801, | |
| 371 | − 0x015021f5, | |
| 372 | − 0xfd050e98, | |
| 373 | − 0x2ebb00ef, | |
| 374 | − 0x003ebb00, | |
| 375 | − 0x98020e98, | |
| 376 | − 0x21f5030f, | |
| 377 | − 0x0e980150, | |
| 378 | − 0x00effd07, | |
| 379 | − 0xbb002ebb, | |
| 380 | − 0x35b6003e, | |
| 381 | − 0x0007f102, | |
| 382 | − 0x0103f0d3, | |
| 383 | − 0xbd0003d0, | |
| 384 | − 0x0825b604, | |
| 385 | − 0xb60635b6, | |
| 386 | − 0x30b60120, | |
| 387 | − 0x0824b601, | |
| 388 | − 0xb90834b6, | |
| 389 | − 0x21f5022f, | |
| 390 | − 0x2fbb02d3, | |
| 391 | − 0x003fbb00, | |
| 392 | − 0x010007f1, | |
| 393 | − 0xd00203f0, | |
| 341 | + 0xe5f050e3, | |
| 342 | + 0xbd24bd01, | |
| 343 | +/* 0x0433: init_unk_loop */ | |
| 344 | + 0xf444bd34, | |
| 345 | + 0xf6b06821, | |
| 346 | + 0x0f0bf400, | |
| 347 | + 0xbb01f7f0, | |
| 348 | + 0x4ffd04f2, | |
| 349 | + 0x0130b605, | |
| 350 | +/* 0x0448: init_unk_next */ | |
| 351 | + 0xb60120b6, | |
| 352 | + 0x26b004e0, | |
| 353 | + 0xe21bf401, | |
| 354 | +/* 0x0454: init_unk_done */ | |
| 355 | + 0x80070380, | |
| 356 | + 0x27f10804, | |
| 357 | + 0x23f00100, | |
| 358 | + 0x0022cf02, | |
| 359 | + 0x259534bd, | |
| 360 | + 0x0007f108, | |
| 361 | + 0x0103f0c0, | |
| 362 | + 0xbd0005d0, | |
| 363 | + 0x0007f104, | |
| 364 | + 0x0103f0c1, | |
| 365 | + 0xbd0005d0, | |
| 366 | + 0x000e9804, | |
| 367 | + 0xf5010f98, | |
| 368 | + 0xbb015021, | |
| 369 | + 0x3fbb002f, | |
| 370 | + 0x010e9800, | |
| 371 | + 0xf5020f98, | |
| 372 | + 0x98015021, | |
| 373 | + 0xeffd050e, | |
| 374 | + 0x002ebb00, | |
| 375 | + 0x98003ebb, | |
| 376 | + 0x0f98020e, | |
| 377 | + 0x5021f503, | |
| 378 | + 0x070e9801, | |
| 379 | + 0xbb00effd, | |
| 380 | + 0x3ebb002e, | |
| 381 | + 0x0235b600, | |
| 382 | + 0xd30007f1, | |
| 383 | + 0xd00103f0, | |
| 394 | 384 | 0x04bd0003, |
| 395 | − 0x29f024bd, | |
| 396 | − 0x0007f11f, | |
| 397 | − 0x0203f008, | |
| 398 | − 0xbd0002d0, | |
| 399 | −/* 0x0505: main */ | |
| 400 | − 0x0031f404, | |
| 401 | − 0xf00028f4, | |
| 402 | − 0x21f424d7, | |
| 403 | − 0xf401f439, | |
| 404 | − 0xf404e4b0, | |
| 405 | − 0x81fe1e18, | |
| 406 | − 0x0627f001, | |
| 407 | − 0x12fd20bd, | |
| 408 | − 0x01e4b604, | |
| 409 | − 0xfe051efd, | |
| 410 | − 0x21f50018, | |
| 411 | − 0x0ef405fa, | |
| 412 | −/* 0x0535: main_not_ctx_xfer */ | |
| 413 | − 0x10ef94d3, | |
| 414 | − 0xf501f5f0, | |
| 415 | − 0xf4037e21, | |
| 416 | −/* 0x0542: ih */ | |
| 417 | − 0x80f9c60e, | |
| 418 | − 0xf90188fe, | |
| 419 | − 0xf990f980, | |
| 420 | − 0xf9b0f9a0, | |
| 421 | − 0xf9e0f9d0, | |
| 422 | − 0xf104bdf0, | |
| 423 | − 0xf00200a7, | |
| 424 | − 0xaacf00a3, | |
| 425 | − 0x04abc400, | |
| 426 | − 0xf02c0bf4, | |
| 427 | − 0xe7f124d7, | |
| 428 | − 0xe3f01a00, | |
| 429 | − 0x00eecf00, | |
| 430 | − 0x1900f7f1, | |
| 431 | − 0xcf00f3f0, | |
| 432 | − 0x21f400ff, | |
| 433 | − 0x01e7f004, | |
| 434 | − 0x1d0007f1, | |
| 435 | − 0xd00003f0, | |
| 436 | − 0x04bd000e, | |
| 437 | −/* 0x0590: ih_no_fifo */ | |
| 438 | − 0x010007f1, | |
| 439 | − 0xd00003f0, | |
| 440 | − 0x04bd000a, | |
| 441 | − 0xe0fcf0fc, | |
| 442 | − 0xb0fcd0fc, | |
| 443 | − 0x90fca0fc, | |
| 444 | − 0x88fe80fc, | |
| 445 | − 0xf480fc00, | |
| 446 | − 0x01f80032, | |
| 447 | −/* 0x05b4: hub_barrier_done */ | |
| 448 | − 0x9801f7f0, | |
| 449 | − 0xfebb040e, | |
| 450 | − 0x02ffb904, | |
| 451 | − 0x9418e7f1, | |
| 452 | − 0xf440e3f0, | |
| 453 | − 0x00f89d21, | |
| 454 | −/* 0x05cc: ctx_redswitch */ | |
| 455 | − 0xf120f7f0, | |
| 385 | + 0xb60825b6, | |
| 386 | + 0x20b60635, | |
| 387 | + 0x0130b601, | |
| 388 | + 0xb60824b6, | |
| 389 | + 0x2fb90834, | |
| 390 | + 0xd321f502, | |
| 391 | + 0x002fbb02, | |
| 392 | + 0xf1003fbb, | |
| 393 | + 0xf0010007, | |
| 394 | + 0x03d00203, | |
| 395 | + 0xbd04bd00, | |
| 396 | + 0x1f29f024, | |
| 397 | + 0x080007f1, | |
| 398 | + 0xd00203f0, | |
| 399 | + 0x04bd0002, | |
| 400 | +/* 0x0508: main */ | |
| 401 | + 0xf40031f4, | |
| 402 | + 0xd7f00028, | |
| 403 | + 0x3921f424, | |
| 404 | + 0xb0f401f4, | |
| 405 | + 0x18f404e4, | |
| 406 | + 0x0181fe1e, | |
| 407 | + 0xbd0627f0, | |
| 408 | + 0x0412fd20, | |
| 409 | + 0xfd01e4b6, | |
| 410 | + 0x18fe051e, | |
| 411 | + 0xfd21f500, | |
| 412 | + 0xd30ef405, | |
| 413 | +/* 0x0538: main_not_ctx_xfer */ | |
| 414 | + 0xf010ef94, | |
| 415 | + 0x21f501f5, | |
| 416 | + 0x0ef4037e, | |
| 417 | +/* 0x0545: ih */ | |
| 418 | + 0xfe80f9c6, | |
| 419 | + 0x80f90188, | |
| 420 | + 0xa0f990f9, | |
| 421 | + 0xd0f9b0f9, | |
| 422 | + 0xf0f9e0f9, | |
| 423 | + 0xa7f104bd, | |
| 424 | + 0xa3f00200, | |
| 425 | + 0x00aacf00, | |
| 426 | + 0xf404abc4, | |
| 427 | + 0xd7f02c0b, | |
| 428 | + 0x00e7f124, | |
| 429 | + 0x00e3f01a, | |
| 430 | + 0xf100eecf, | |
| 431 | + 0xf01900f7, | |
| 432 | + 0xffcf00f3, | |
| 433 | + 0x0421f400, | |
| 434 | + 0xf101e7f0, | |
| 435 | + 0xf01d0007, | |
| 436 | + 0x0ed00003, | |
| 437 | +/* 0x0593: ih_no_fifo */ | |
| 438 | + 0xf104bd00, | |
| 439 | + 0xf0010007, | |
| 440 | + 0x0ad00003, | |
| 441 | + 0xfc04bd00, | |
| 442 | + 0xfce0fcf0, | |
| 443 | + 0xfcb0fcd0, | |
| 444 | + 0xfc90fca0, | |
| 445 | + 0x0088fe80, | |
| 446 | + 0x32f480fc, | |
| 447 | +/* 0x05b7: hub_barrier_done */ | |
| 448 | + 0xf001f800, | |
| 449 | + 0x0e9801f7, | |
| 450 | + 0x04febb04, | |
| 451 | + 0xf102ffb9, | |
| 452 | + 0xf09418e7, | |
| 453 | + 0x21f440e3, | |
| 454 | +/* 0x05cf: ctx_redswitch */ | |
| 455 | + 0xf000f89d, | |
| 456 | + 0x07f120f7, | |
| 457 | + 0x03f08500, | |
| 458 | + 0x000fd001, | |
| 459 | + 0xe7f004bd, | |
| 460 | +/* 0x05e1: ctx_redswitch_delay */ | |
| 461 | + 0x01e2b608, | |
| 462 | + 0xf1fd1bf4, | |
| 463 | + 0xf10800f5, | |
| 464 | + 0xf10200f5, | |
| 456 | 465 | 0xf0850007, |
| 457 | 466 | 0x0fd00103, |
| 458 | − 0xf004bd00, | |
| 459 | −/* 0x05de: ctx_redswitch_delay */ | |
| 460 | − 0xe2b608e7, | |
| 461 | − 0xfd1bf401, | |
| 462 | − 0x0800f5f1, | |
| 463 | − 0x0200f5f1, | |
| 464 | − 0x850007f1, | |
| 465 | − 0xd00103f0, | |
| 466 | − 0x04bd000f, | |
| 467 | −/* 0x05fa: ctx_xfer */ | |
| 468 | − 0x07f100f8, | |
| 469 | − 0x03f08100, | |
| 470 | − 0x000fd002, | |
| 471 | − 0x11f404bd, | |
| 472 | − 0xcc21f507, | |
| 473 | −/* 0x060d: ctx_xfer_not_load */ | |
| 474 | − 0x6a21f505, | |
| 475 | − 0xf124bd02, | |
| 476 | − 0xf047fc07, | |
| 477 | − 0x02d00203, | |
| 478 | − 0xf004bd00, | |
| 479 | − 0x20b6012c, | |
| 480 | − 0xfc07f103, | |
| 481 | − 0x0203f04a, | |
| 482 | − 0xbd0002d0, | |
| 483 | − 0x01acf004, | |
| 484 | − 0xf102a5f0, | |
| 485 | − 0xf00000b7, | |
| 486 | − 0x0c9850b3, | |
| 487 | − 0x0fc4b604, | |
| 488 | − 0x9800bcbb, | |
| 489 | − 0x0d98000c, | |
| 490 | − 0x00e7f001, | |
| 491 | − 0x016f21f5, | |
| 492 | − 0xf101acf0, | |
| 493 | − 0xf04000b7, | |
| 494 | − 0x0c9850b3, | |
| 495 | − 0x0fc4b604, | |
| 496 | − 0x9800bcbb, | |
| 497 | − 0x0d98010c, | |
| 498 | − 0x060f9802, | |
| 499 | − 0x0800e7f1, | |
| 500 | − 0x016f21f5, | |
| 467 | + 0xf804bd00, | |
| 468 | +/* 0x05fd: ctx_xfer */ | |
| 469 | + 0x0007f100, | |
| 470 | + 0x0203f081, | |
| 471 | + 0xbd000fd0, | |
| 472 | + 0x0711f404, | |
| 473 | + 0x05cf21f5, | |
| 474 | +/* 0x0610: ctx_xfer_not_load */ | |
| 475 | + 0x026a21f5, | |
| 476 | + 0x07f124bd, | |
| 477 | + 0x03f047fc, | |
| 478 | + 0x0002d002, | |
| 479 | + 0x2cf004bd, | |
| 480 | + 0x0320b601, | |
| 481 | + 0x4afc07f1, | |
| 482 | + 0xd00203f0, | |
| 483 | + 0x04bd0002, | |
| 501 | 484 | 0xf001acf0, |
| 502 | − 0xb7f104a5, | |
| 503 | − 0xb3f03000, | |
| 485 | + 0xb7f102a5, | |
| 486 | + 0xb3f00000, | |
| 504 | 487 | 0x040c9850, |
| 505 | 488 | 0xbb0fc4b6, |
| 506 | 489 | 0x0c9800bc, |
| 507 | − 0x030d9802, | |
| 508 | − 0xf1080f98, | |
| 509 | − 0xf50200e7, | |
| 510 | − 0xf5016f21, | |
| 511 | − 0xf4025e21, | |
| 512 | − 0x12f40601, | |
| 513 | −/* 0x06a9: ctx_xfer_post */ | |
| 514 | − 0x7f21f507, | |
| 515 | −/* 0x06ad: ctx_xfer_done */ | |
| 516 | − 0xb421f502, | |
| 517 | − 0x0000f805, | |
| 518 | − 0x00000000, | |
| 490 | + 0x010d9800, | |
| 491 | + 0xf500e7f0, | |
| 492 | + 0xf0016f21, | |
| 493 | + 0xb7f101ac, | |
| 494 | + 0xb3f04000, | |
| 495 | + 0x040c9850, | |
| 496 | + 0xbb0fc4b6, | |
| 497 | + 0x0c9800bc, | |
| 498 | + 0x020d9801, | |
| 499 | + 0xf1060f98, | |
| 500 | + 0xf50800e7, | |
| 501 | + 0xf0016f21, | |
| 502 | + 0xa5f001ac, | |
| 503 | + 0x00b7f104, | |
| 504 | + 0x50b3f030, | |
| 505 | + 0xb6040c98, | |
| 506 | + 0xbcbb0fc4, | |
| 507 | + 0x020c9800, | |
| 508 | + 0x98030d98, | |
| 509 | + 0xe7f1080f, | |
| 510 | + 0x21f50200, | |
| 511 | + 0x21f5016f, | |
| 512 | + 0x01f4025e, | |
| 513 | + 0x0712f406, | |
| 514 | +/* 0x06ac: ctx_xfer_post */ | |
| 515 | + 0x027f21f5, | |
| 516 | +/* 0x06b0: ctx_xfer_done */ | |
| 517 | + 0x05b721f5, | |
| 518 | + 0x000000f8, | |
| 519 | 519 | 0x00000000, |
| 520 | 520 | 0x00000000, |
| 521 | 521 | 0x00000000, |
| @@ -314,7 +314,7 @@ uint32_t gk104_grgpc_code[] = { · 313 unmodified lines | ||
| 314 | 314 | 0x03f01200, |
| 315 | 315 | 0x0002d000, |
| 316 | 316 | 0x17f104bd, |
| 317 | − 0x10fe0542, | |
| 317 | + 0x10fe0545, | |
| 318 | 318 | 0x0007f100, |
| 319 | 319 | 0x0003f007, |
| 320 | 320 | 0xbd0000d0, |
| @@ -338,184 +338,184 @@ uint32_t gk104_grgpc_code[] = { · 17 unmodified lines | ||
| 338 | 338 | 0x02d00103, |
| 339 | 339 | 0xf104bd00, |
| 340 | 340 | 0xf00c30e7, |
| 341 | − 0x24bd50e3, | |
| 342 | − 0x44bd34bd, | |
| 343 | −/* 0x0430: init_unk_loop */ | |
| 344 | − 0xb06821f4, | |
| 345 | − 0x0bf400f6, | |
| 346 | − 0x01f7f00f, | |
| 347 | − 0xfd04f2bb, | |
| 348 | − 0x30b6054f, | |
| 349 | −/* 0x0445: init_unk_next */ | |
| 350 | − 0x0120b601, | |
| 351 | − 0xb004e0b6, | |
| 352 | − 0x1bf40126, | |
| 353 | −/* 0x0451: init_unk_done */ | |
| 354 | − 0x070380e2, | |
| 355 | − 0xf1080480, | |
| 356 | − 0xf0010027, | |
| 357 | − 0x22cf0223, | |
| 358 | − 0x9534bd00, | |
| 359 | − 0x07f10825, | |
| 360 | − 0x03f0c000, | |
| 361 | − 0x0005d001, | |
| 362 | − 0x07f104bd, | |
| 363 | − 0x03f0c100, | |
| 364 | − 0x0005d001, | |
| 365 | − 0x0e9804bd, | |
| 366 | − 0x010f9800, | |
| 367 | − 0x015021f5, | |
| 368 | − 0xbb002fbb, | |
| 369 | − 0x0e98003f, | |
| 370 | − 0x020f9801, | |
| 371 | − 0x015021f5, | |
| 372 | − 0xfd050e98, | |
| 373 | − 0x2ebb00ef, | |
| 374 | − 0x003ebb00, | |
| 375 | − 0x98020e98, | |
| 376 | − 0x21f5030f, | |
| 377 | − 0x0e980150, | |
| 378 | − 0x00effd07, | |
| 379 | − 0xbb002ebb, | |
| 380 | − 0x35b6003e, | |
| 381 | − 0x0007f102, | |
| 382 | − 0x0103f0d3, | |
| 383 | − 0xbd0003d0, | |
| 384 | − 0x0825b604, | |
| 385 | − 0xb60635b6, | |
| 386 | − 0x30b60120, | |
| 387 | − 0x0824b601, | |
| 388 | − 0xb90834b6, | |
| 389 | − 0x21f5022f, | |
| 390 | − 0x2fbb02d3, | |
| 391 | − 0x003fbb00, | |
| 392 | − 0x010007f1, | |
| 393 | − 0xd00203f0, | |
| 341 | + 0xe5f050e3, | |
| 342 | + 0xbd24bd01, | |
| 343 | +/* 0x0433: init_unk_loop */ | |
| 344 | + 0xf444bd34, | |
| 345 | + 0xf6b06821, | |
| 346 | + 0x0f0bf400, | |
| 347 | + 0xbb01f7f0, | |
| 348 | + 0x4ffd04f2, | |
| 349 | + 0x0130b605, | |
| 350 | +/* 0x0448: init_unk_next */ | |
| 351 | + 0xb60120b6, | |
| 352 | + 0x26b004e0, | |
| 353 | + 0xe21bf401, | |
| 354 | +/* 0x0454: init_unk_done */ | |
| 355 | + 0x80070380, | |
| 356 | + 0x27f10804, | |
| 357 | + 0x23f00100, | |
| 358 | + 0x0022cf02, | |
| 359 | + 0x259534bd, | |
| 360 | + 0x0007f108, | |
| 361 | + 0x0103f0c0, | |
| 362 | + 0xbd0005d0, | |
| 363 | + 0x0007f104, | |
| 364 | + 0x0103f0c1, | |
| 365 | + 0xbd0005d0, | |
| 366 | + 0x000e9804, | |
| 367 | + 0xf5010f98, | |
| 368 | + 0xbb015021, | |
| 369 | + 0x3fbb002f, | |
| 370 | + 0x010e9800, | |
| 371 | + 0xf5020f98, | |
| 372 | + 0x98015021, | |
| 373 | + 0xeffd050e, | |
| 374 | + 0x002ebb00, | |
| 375 | + 0x98003ebb, | |
| 376 | + 0x0f98020e, | |
| 377 | + 0x5021f503, | |
| 378 | + 0x070e9801, | |
| 379 | + 0xbb00effd, | |
| 380 | + 0x3ebb002e, | |
| 381 | + 0x0235b600, | |
| 382 | + 0xd30007f1, | |
| 383 | + 0xd00103f0, | |
| 394 | 384 | 0x04bd0003, |
| 395 | − 0x29f024bd, | |
| 396 | − 0x0007f11f, | |
| 397 | − 0x0203f008, | |
| 398 | − 0xbd0002d0, | |
| 399 | −/* 0x0505: main */ | |
| 400 | − 0x0031f404, | |
| 401 | − 0xf00028f4, | |
| 402 | − 0x21f424d7, | |
| 403 | − 0xf401f439, | |
| 404 | − 0xf404e4b0, | |
| 405 | − 0x81fe1e18, | |
| 406 | − 0x0627f001, | |
| 407 | − 0x12fd20bd, | |
| 408 | − 0x01e4b604, | |
| 409 | − 0xfe051efd, | |
| 410 | − 0x21f50018, | |
| 411 | − 0x0ef405fa, | |
| 412 | −/* 0x0535: main_not_ctx_xfer */ | |
| 413 | − 0x10ef94d3, | |
| 414 | − 0xf501f5f0, | |
| 415 | − 0xf4037e21, | |
| 416 | −/* 0x0542: ih */ | |
| 417 | − 0x80f9c60e, | |
| 418 | − 0xf90188fe, | |
| 419 | − 0xf990f980, | |
| 420 | − 0xf9b0f9a0, | |
| 421 | − 0xf9e0f9d0, | |
| 422 | − 0xf104bdf0, | |
| 423 | − 0xf00200a7, | |
| 424 | − 0xaacf00a3, | |
| 425 | − 0x04abc400, | |
| 426 | − 0xf02c0bf4, | |
| 427 | − 0xe7f124d7, | |
| 428 | − 0xe3f01a00, | |
| 429 | − 0x00eecf00, | |
| 430 | − 0x1900f7f1, | |
| 431 | − 0xcf00f3f0, | |
| 432 | − 0x21f400ff, | |
| 433 | − 0x01e7f004, | |
| 434 | − 0x1d0007f1, | |
| 435 | − 0xd00003f0, | |
| 436 | − 0x04bd000e, | |
| 437 | −/* 0x0590: ih_no_fifo */ | |
| 438 | − 0x010007f1, | |
| 439 | − 0xd00003f0, | |
| 440 | − 0x04bd000a, | |
| 441 | − 0xe0fcf0fc, | |
| 442 | − 0xb0fcd0fc, | |
| 443 | − 0x90fca0fc, | |
| 444 | − 0x88fe80fc, | |
| 445 | − 0xf480fc00, | |
| 446 | − 0x01f80032, | |
| 447 | −/* 0x05b4: hub_barrier_done */ | |
| 448 | − 0x9801f7f0, | |
| 449 | − 0xfebb040e, | |
| 450 | − 0x02ffb904, | |
| 451 | − 0x9418e7f1, | |
| 452 | − 0xf440e3f0, | |
| 453 | − 0x00f89d21, | |
| 454 | −/* 0x05cc: ctx_redswitch */ | |
| 455 | − 0xf120f7f0, | |
| 385 | + 0xb60825b6, | |
| 386 | + 0x20b60635, | |
| 387 | + 0x0130b601, | |
| 388 | + 0xb60824b6, | |
| 389 | + 0x2fb90834, | |
| 390 | + 0xd321f502, | |
| 391 | + 0x002fbb02, | |
| 392 | + 0xf1003fbb, | |
| 393 | + 0xf0010007, | |
| 394 | + 0x03d00203, | |
| 395 | + 0xbd04bd00, | |
| 396 | + 0x1f29f024, | |
| 397 | + 0x080007f1, | |
| 398 | + 0xd00203f0, | |
| 399 | + 0x04bd0002, | |
| 400 | +/* 0x0508: main */ | |
| 401 | + 0xf40031f4, | |
| 402 | + 0xd7f00028, | |
| 403 | + 0x3921f424, | |
| 404 | + 0xb0f401f4, | |
| 405 | + 0x18f404e4, | |
| 406 | + 0x0181fe1e, | |
| 407 | + 0xbd0627f0, | |
| 408 | + 0x0412fd20, | |
| 409 | + 0xfd01e4b6, | |
| 410 | + 0x18fe051e, | |
| 411 | + 0xfd21f500, | |
| 412 | + 0xd30ef405, | |
| 413 | +/* 0x0538: main_not_ctx_xfer */ | |
| 414 | + 0xf010ef94, | |
| 415 | + 0x21f501f5, | |
| 416 | + 0x0ef4037e, | |
| 417 | +/* 0x0545: ih */ | |
| 418 | + 0xfe80f9c6, | |
| 419 | + 0x80f90188, | |
| 420 | + 0xa0f990f9, | |
| 421 | + 0xd0f9b0f9, | |
| 422 | + 0xf0f9e0f9, | |
| 423 | + 0xa7f104bd, | |
| 424 | + 0xa3f00200, | |
| 425 | + 0x00aacf00, | |
| 426 | + 0xf404abc4, | |
| 427 | + 0xd7f02c0b, | |
| 428 | + 0x00e7f124, | |
| 429 | + 0x00e3f01a, | |
| 430 | + 0xf100eecf, | |
| 431 | + 0xf01900f7, | |
| 432 | + 0xffcf00f3, | |
| 433 | + 0x0421f400, | |
| 434 | + 0xf101e7f0, | |
| 435 | + 0xf01d0007, | |
| 436 | + 0x0ed00003, | |
| 437 | +/* 0x0593: ih_no_fifo */ | |
| 438 | + 0xf104bd00, | |
| 439 | + 0xf0010007, | |
| 440 | + 0x0ad00003, | |
| 441 | + 0xfc04bd00, | |
| 442 | + 0xfce0fcf0, | |
| 443 | + 0xfcb0fcd0, | |
| 444 | + 0xfc90fca0, | |
| 445 | + 0x0088fe80, | |
| 446 | + 0x32f480fc, | |
| 447 | +/* 0x05b7: hub_barrier_done */ | |
| 448 | + 0xf001f800, | |
| 449 | + 0x0e9801f7, | |
| 450 | + 0x04febb04, | |
| 451 | + 0xf102ffb9, | |
| 452 | + 0xf09418e7, | |
| 453 | + 0x21f440e3, | |
| 454 | +/* 0x05cf: ctx_redswitch */ | |
| 455 | + 0xf000f89d, | |
| 456 | + 0x07f120f7, | |
| 457 | + 0x03f08500, | |
| 458 | + 0x000fd001, | |
| 459 | + 0xe7f004bd, | |
| 460 | +/* 0x05e1: ctx_redswitch_delay */ | |
| 461 | + 0x01e2b608, | |
| 462 | + 0xf1fd1bf4, | |
| 463 | + 0xf10800f5, | |
| 464 | + 0xf10200f5, | |
| 456 | 465 | 0xf0850007, |
| 457 | 466 | 0x0fd00103, |
| 458 | − 0xf004bd00, | |
| 459 | −/* 0x05de: ctx_redswitch_delay */ | |
| 460 | − 0xe2b608e7, | |
| 461 | − 0xfd1bf401, | |
| 462 | − 0x0800f5f1, | |
| 463 | − 0x0200f5f1, | |
| 464 | − 0x850007f1, | |
| 465 | − 0xd00103f0, | |
| 466 | − 0x04bd000f, | |
| 467 | −/* 0x05fa: ctx_xfer */ | |
| 468 | − 0x07f100f8, | |
| 469 | − 0x03f08100, | |
| 470 | − 0x000fd002, | |
| 471 | − 0x11f404bd, | |
| 472 | − 0xcc21f507, | |
| 473 | −/* 0x060d: ctx_xfer_not_load */ | |
| 474 | − 0x6a21f505, | |
| 475 | − 0xf124bd02, | |
| 476 | − 0xf047fc07, | |
| 477 | − 0x02d00203, | |
| 478 | − 0xf004bd00, | |
| 479 | − 0x20b6012c, | |
| 480 | − 0xfc07f103, | |
| 481 | − 0x0203f04a, | |
| 482 | − 0xbd0002d0, | |
| 483 | − 0x01acf004, | |
| 484 | − 0xf102a5f0, | |
| 485 | − 0xf00000b7, | |
| 486 | − 0x0c9850b3, | |
| 487 | − 0x0fc4b604, | |
| 488 | − 0x9800bcbb, | |
| 489 | − 0x0d98000c, | |
| 490 | − 0x00e7f001, | |
| 491 | − 0x016f21f5, | |
| 492 | − 0xf101acf0, | |
| 493 | − 0xf04000b7, | |
| 494 | − 0x0c9850b3, | |
| 495 | − 0x0fc4b604, | |
| 496 | − 0x9800bcbb, | |
| 497 | − 0x0d98010c, | |
| 498 | − 0x060f9802, | |
| 499 | − 0x0800e7f1, | |
| 500 | − 0x016f21f5, | |
| 467 | + 0xf804bd00, | |
| 468 | +/* 0x05fd: ctx_xfer */ | |
| 469 | + 0x0007f100, | |
| 470 | + 0x0203f081, | |
| 471 | + 0xbd000fd0, | |
| 472 | + 0x0711f404, | |
| 473 | + 0x05cf21f5, | |
| 474 | +/* 0x0610: ctx_xfer_not_load */ | |
| 475 | + 0x026a21f5, | |
| 476 | + 0x07f124bd, | |
| 477 | + 0x03f047fc, | |
| 478 | + 0x0002d002, | |
| 479 | + 0x2cf004bd, | |
| 480 | + 0x0320b601, | |
| 481 | + 0x4afc07f1, | |
| 482 | + 0xd00203f0, | |
| 483 | + 0x04bd0002, | |
| 501 | 484 | 0xf001acf0, |
| 502 | − 0xb7f104a5, | |
| 503 | − 0xb3f03000, | |
| 485 | + 0xb7f102a5, | |
| 486 | + 0xb3f00000, | |
| 504 | 487 | 0x040c9850, |
| 505 | 488 | 0xbb0fc4b6, |
| 506 | 489 | 0x0c9800bc, |
| 507 | − 0x030d9802, | |
| 508 | − 0xf1080f98, | |
| 509 | − 0xf50200e7, | |
| 510 | − 0xf5016f21, | |
| 511 | − 0xf4025e21, | |
| 512 | − 0x12f40601, | |
| 513 | −/* 0x06a9: ctx_xfer_post */ | |
| 514 | − 0x7f21f507, | |
| 515 | −/* 0x06ad: ctx_xfer_done */ | |
| 516 | − 0xb421f502, | |
| 517 | − 0x0000f805, | |
| 518 | − 0x00000000, | |
| 490 | + 0x010d9800, | |
| 491 | + 0xf500e7f0, | |
| 492 | + 0xf0016f21, | |
| 493 | + 0xb7f101ac, | |
| 494 | + 0xb3f04000, | |
| 495 | + 0x040c9850, | |
| 496 | + 0xbb0fc4b6, | |
| 497 | + 0x0c9800bc, | |
| 498 | + 0x020d9801, | |
| 499 | + 0xf1060f98, | |
| 500 | + 0xf50800e7, | |
| 501 | + 0xf0016f21, | |
| 502 | + 0xa5f001ac, | |
| 503 | + 0x00b7f104, | |
| 504 | + 0x50b3f030, | |
| 505 | + 0xb6040c98, | |
| 506 | + 0xbcbb0fc4, | |
| 507 | + 0x020c9800, | |
| 508 | + 0x98030d98, | |
| 509 | + 0xe7f1080f, | |
| 510 | + 0x21f50200, | |
| 511 | + 0x21f5016f, | |
| 512 | + 0x01f4025e, | |
| 513 | + 0x0712f406, | |
| 514 | +/* 0x06ac: ctx_xfer_post */ | |
| 515 | + 0x027f21f5, | |
| 516 | +/* 0x06b0: ctx_xfer_done */ | |
| 517 | + 0x05b721f5, | |
| 518 | + 0x000000f8, | |
| 519 | 519 | 0x00000000, |
| 520 | 520 | 0x00000000, |
| 521 | 521 | 0x00000000, |
| @@ -314,7 +314,7 @@ uint32_t gk110_grgpc_code[] = { · 313 unmodified lines | ||
| 314 | 314 | 0x03f01200, |
| 315 | 315 | 0x0002d000, |
| 316 | 316 | 0x17f104bd, |
| 317 | − 0x10fe0542, | |
| 317 | + 0x10fe0545, | |
| 318 | 318 | 0x0007f100, |
| 319 | 319 | 0x0003f007, |
| 320 | 320 | 0xbd0000d0, |
| @@ -338,184 +338,184 @@ uint32_t gk110_grgpc_code[] = { · 17 unmodified lines | ||
| 338 | 338 | 0x02d00103, |
| 339 | 339 | 0xf104bd00, |
| 340 | 340 | 0xf00c30e7, |
| 341 | − 0x24bd50e3, | |
| 342 | − 0x44bd34bd, | |
| 343 | −/* 0x0430: init_unk_loop */ | |
| 344 | − 0xb06821f4, | |
| 345 | − 0x0bf400f6, | |
| 346 | − 0x01f7f00f, | |
| 347 | − 0xfd04f2bb, | |
| 348 | − 0x30b6054f, | |
| 349 | −/* 0x0445: init_unk_next */ | |
| 350 | − 0x0120b601, | |
| 351 | − 0xb004e0b6, | |
| 352 | − 0x1bf40226, | |
| 353 | −/* 0x0451: init_unk_done */ | |
| 354 | − 0x070380e2, | |
| 355 | − 0xf1080480, | |
| 356 | − 0xf0010027, | |
| 357 | − 0x22cf0223, | |
| 358 | − 0x9534bd00, | |
| 359 | − 0x07f10825, | |
| 360 | − 0x03f0c000, | |
| 361 | − 0x0005d001, | |
| 362 | − 0x07f104bd, | |
| 363 | − 0x03f0c100, | |
| 364 | − 0x0005d001, | |
| 365 | − 0x0e9804bd, | |
| 366 | − 0x010f9800, | |
| 367 | − 0x015021f5, | |
| 368 | − 0xbb002fbb, | |
| 369 | − 0x0e98003f, | |
| 370 | − 0x020f9801, | |
| 371 | − 0x015021f5, | |
| 372 | − 0xfd050e98, | |
| 373 | − 0x2ebb00ef, | |
| 374 | − 0x003ebb00, | |
| 375 | − 0x98020e98, | |
| 376 | − 0x21f5030f, | |
| 377 | − 0x0e980150, | |
| 378 | − 0x00effd07, | |
| 379 | − 0xbb002ebb, | |
| 380 | − 0x35b6003e, | |
| 381 | − 0x0007f102, | |
| 382 | − 0x0103f0d3, | |
| 383 | − 0xbd0003d0, | |
| 384 | − 0x0825b604, | |
| 385 | − 0xb60635b6, | |
| 386 | − 0x30b60120, | |
| 387 | − 0x0824b601, | |
| 388 | − 0xb90834b6, | |
| 389 | − 0x21f5022f, | |
| 390 | − 0x2fbb02d3, | |
| 391 | − 0x003fbb00, | |
| 392 | − 0x010007f1, | |
| 393 | − 0xd00203f0, | |
| 341 | + 0xe5f050e3, | |
| 342 | + 0xbd24bd01, | |
| 343 | +/* 0x0433: init_unk_loop */ | |
| 344 | + 0xf444bd34, | |
| 345 | + 0xf6b06821, | |
| 346 | + 0x0f0bf400, | |
| 347 | + 0xbb01f7f0, | |
| 348 | + 0x4ffd04f2, | |
| 349 | + 0x0130b605, | |
| 350 | +/* 0x0448: init_unk_next */ | |
| 351 | + 0xb60120b6, | |
| 352 | + 0x26b004e0, | |
| 353 | + 0xe21bf402, | |
| 354 | +/* 0x0454: init_unk_done */ | |
| 355 | + 0x80070380, | |
| 356 | + 0x27f10804, | |
| 357 | + 0x23f00100, | |
| 358 | + 0x0022cf02, | |
| 359 | + 0x259534bd, | |
| 360 | + 0x0007f108, | |
| 361 | + 0x0103f0c0, | |
| 362 | + 0xbd0005d0, | |
| 363 | + 0x0007f104, | |
| 364 | + 0x0103f0c1, | |
| 365 | + 0xbd0005d0, | |
| 366 | + 0x000e9804, | |
| 367 | + 0xf5010f98, | |
| 368 | + 0xbb015021, | |
| 369 | + 0x3fbb002f, | |
| 370 | + 0x010e9800, | |
| 371 | + 0xf5020f98, | |
| 372 | + 0x98015021, | |
| 373 | + 0xeffd050e, | |
| 374 | + 0x002ebb00, | |
| 375 | + 0x98003ebb, | |
| 376 | + 0x0f98020e, | |
| 377 | + 0x5021f503, | |
| 378 | + 0x070e9801, | |
| 379 | + 0xbb00effd, | |
| 380 | + 0x3ebb002e, | |
| 381 | + 0x0235b600, | |
| 382 | + 0xd30007f1, | |
| 383 | + 0xd00103f0, | |
| 394 | 384 | 0x04bd0003, |
| 395 | − 0x29f024bd, | |
| 396 | − 0x0007f11f, | |
| 397 | − 0x0203f030, | |
| 398 | − 0xbd0002d0, | |
| 399 | −/* 0x0505: main */ | |
| 400 | − 0x0031f404, | |
| 401 | − 0xf00028f4, | |
| 402 | − 0x21f424d7, | |
| 403 | − 0xf401f439, | |
| 404 | − 0xf404e4b0, | |
| 405 | − 0x81fe1e18, | |
| 406 | − 0x0627f001, | |
| 407 | − 0x12fd20bd, | |
| 408 | − 0x01e4b604, | |
| 409 | − 0xfe051efd, | |
| 410 | − 0x21f50018, | |
| 411 | − 0x0ef405fa, | |
| 412 | −/* 0x0535: main_not_ctx_xfer */ | |
| 413 | − 0x10ef94d3, | |
| 414 | − 0xf501f5f0, | |
| 415 | − 0xf4037e21, | |
| 416 | −/* 0x0542: ih */ | |
| 417 | − 0x80f9c60e, | |
| 418 | − 0xf90188fe, | |
| 419 | − 0xf990f980, | |
| 420 | − 0xf9b0f9a0, | |
| 421 | − 0xf9e0f9d0, | |
| 422 | − 0xf104bdf0, | |
| 423 | − 0xf00200a7, | |
| 424 | − 0xaacf00a3, | |
| 425 | − 0x04abc400, | |
| 426 | − 0xf02c0bf4, | |
| 427 | − 0xe7f124d7, | |
| 428 | − 0xe3f01a00, | |
| 429 | − 0x00eecf00, | |
| 430 | − 0x1900f7f1, | |
| 431 | − 0xcf00f3f0, | |
| 432 | − 0x21f400ff, | |
| 433 | − 0x01e7f004, | |
| 434 | − 0x1d0007f1, | |
| 435 | − 0xd00003f0, | |
| 436 | − 0x04bd000e, | |
| 437 | −/* 0x0590: ih_no_fifo */ | |
| 438 | − 0x010007f1, | |
| 439 | − 0xd00003f0, | |
| 440 | − 0x04bd000a, | |
| 441 | − 0xe0fcf0fc, | |
| 442 | − 0xb0fcd0fc, | |
| 443 | − 0x90fca0fc, | |
| 444 | − 0x88fe80fc, | |
| 445 | − 0xf480fc00, | |
| 446 | − 0x01f80032, | |
| 447 | −/* 0x05b4: hub_barrier_done */ | |
| 448 | − 0x9801f7f0, | |
| 449 | − 0xfebb040e, | |
| 450 | − 0x02ffb904, | |
| 451 | − 0x9418e7f1, | |
| 452 | − 0xf440e3f0, | |
| 453 | − 0x00f89d21, | |
| 454 | −/* 0x05cc: ctx_redswitch */ | |
| 455 | − 0xf120f7f0, | |
| 385 | + 0xb60825b6, | |
| 386 | + 0x20b60635, | |
| 387 | + 0x0130b601, | |
| 388 | + 0xb60824b6, | |
| 389 | + 0x2fb90834, | |
| 390 | + 0xd321f502, | |
| 391 | + 0x002fbb02, | |
| 392 | + 0xf1003fbb, | |
| 393 | + 0xf0010007, | |
| 394 | + 0x03d00203, | |
| 395 | + 0xbd04bd00, | |
| 396 | + 0x1f29f024, | |
| 397 | + 0x300007f1, | |
| 398 | + 0xd00203f0, | |
| 399 | + 0x04bd0002, | |
| 400 | +/* 0x0508: main */ | |
| 401 | + 0xf40031f4, | |
| 402 | + 0xd7f00028, | |
| 403 | + 0x3921f424, | |
| 404 | + 0xb0f401f4, | |
| 405 | + 0x18f404e4, | |
| 406 | + 0x0181fe1e, | |
| 407 | + 0xbd0627f0, | |
| 408 | + 0x0412fd20, | |
| 409 | + 0xfd01e4b6, | |
| 410 | + 0x18fe051e, | |
| 411 | + 0xfd21f500, | |
| 412 | + 0xd30ef405, | |
| 413 | +/* 0x0538: main_not_ctx_xfer */ | |
| 414 | + 0xf010ef94, | |
| 415 | + 0x21f501f5, | |
| 416 | + 0x0ef4037e, | |
| 417 | +/* 0x0545: ih */ | |
| 418 | + 0xfe80f9c6, | |
| 419 | + 0x80f90188, | |
| 420 | + 0xa0f990f9, | |
| 421 | + 0xd0f9b0f9, | |
| 422 | + 0xf0f9e0f9, | |
| 423 | + 0xa7f104bd, | |
| 424 | + 0xa3f00200, | |
| 425 | + 0x00aacf00, | |
| 426 | + 0xf404abc4, | |
| 427 | + 0xd7f02c0b, | |
| 428 | + 0x00e7f124, | |
| 429 | + 0x00e3f01a, | |
| 430 | + 0xf100eecf, | |
| 431 | + 0xf01900f7, | |
| 432 | + 0xffcf00f3, | |
| 433 | + 0x0421f400, | |
| 434 | + 0xf101e7f0, | |
| 435 | + 0xf01d0007, | |
| 436 | + 0x0ed00003, | |
| 437 | +/* 0x0593: ih_no_fifo */ | |
| 438 | + 0xf104bd00, | |
| 439 | + 0xf0010007, | |
| 440 | + 0x0ad00003, | |
| 441 | + 0xfc04bd00, | |
| 442 | + 0xfce0fcf0, | |
| 443 | + 0xfcb0fcd0, | |
| 444 | + 0xfc90fca0, | |
| 445 | + 0x0088fe80, | |
| 446 | + 0x32f480fc, | |
| 447 | +/* 0x05b7: hub_barrier_done */ | |
| 448 | + 0xf001f800, | |
| 449 | + 0x0e9801f7, | |
| 450 | + 0x04febb04, | |
| 451 | + 0xf102ffb9, | |
| 452 | + 0xf09418e7, | |
| 453 | + 0x21f440e3, | |
| 454 | +/* 0x05cf: ctx_redswitch */ | |
| 455 | + 0xf000f89d, | |
| 456 | + 0x07f120f7, | |
| 457 | + 0x03f08500, | |
| 458 | + 0x000fd001, | |
| 459 | + 0xe7f004bd, | |
| 460 | +/* 0x05e1: ctx_redswitch_delay */ | |
| 461 | + 0x01e2b608, | |
| 462 | + 0xf1fd1bf4, | |
| 463 | + 0xf10800f5, | |
| 464 | + 0xf10200f5, | |
| 456 | 465 | 0xf0850007, |
| 457 | 466 | 0x0fd00103, |
| 458 | − 0xf004bd00, | |
| 459 | −/* 0x05de: ctx_redswitch_delay */ | |
| 460 | − 0xe2b608e7, | |
| 461 | − 0xfd1bf401, | |
| 462 | − 0x0800f5f1, | |
| 463 | − 0x0200f5f1, | |
| 464 | − 0x850007f1, | |
| 465 | − 0xd00103f0, | |
| 466 | − 0x04bd000f, | |
| 467 | −/* 0x05fa: ctx_xfer */ | |
| 468 | − 0x07f100f8, | |
| 469 | − 0x03f08100, | |
| 470 | − 0x000fd002, | |
| 471 | − 0x11f404bd, | |
| 472 | − 0xcc21f507, | |
| 473 | −/* 0x060d: ctx_xfer_not_load */ | |
| 474 | − 0x6a21f505, | |
| 475 | − 0xf124bd02, | |
| 476 | − 0xf047fc07, | |
| 477 | − 0x02d00203, | |
| 478 | − 0xf004bd00, | |
| 479 | − 0x20b6012c, | |
| 480 | − 0xfc07f103, | |
| 481 | − 0x0203f04a, | |
| 482 | − 0xbd0002d0, | |
| 483 | − 0x01acf004, | |
| 484 | − 0xf102a5f0, | |
| 485 | − 0xf00000b7, | |
| 486 | − 0x0c9850b3, | |
| 487 | − 0x0fc4b604, | |
| 488 | − 0x9800bcbb, | |
| 489 | − 0x0d98000c, | |
| 490 | − 0x00e7f001, | |
| 491 | − 0x016f21f5, | |
| 492 | − 0xf101acf0, | |
| 493 | − 0xf04000b7, | |
| 494 | − 0x0c9850b3, | |
| 495 | − 0x0fc4b604, | |
| 496 | − 0x9800bcbb, | |
| 497 | − 0x0d98010c, | |
| 498 | − 0x060f9802, | |
| 499 | − 0x0800e7f1, | |
| 500 | − 0x016f21f5, | |
| 467 | + 0xf804bd00, | |
| 468 | +/* 0x05fd: ctx_xfer */ | |
| 469 | + 0x0007f100, | |
| 470 | + 0x0203f081, | |
| 471 | + 0xbd000fd0, | |
| 472 | + 0x0711f404, | |
| 473 | + 0x05cf21f5, | |
| 474 | +/* 0x0610: ctx_xfer_not_load */ | |
| 475 | + 0x026a21f5, | |
| 476 | + 0x07f124bd, | |
| 477 | + 0x03f047fc, | |
| 478 | + 0x0002d002, | |
| 479 | + 0x2cf004bd, | |
| 480 | + 0x0320b601, | |
| 481 | + 0x4afc07f1, | |
| 482 | + 0xd00203f0, | |
| 483 | + 0x04bd0002, | |
| 501 | 484 | 0xf001acf0, |
| 502 | − 0xb7f104a5, | |
| 503 | − 0xb3f03000, | |
| 485 | + 0xb7f102a5, | |
| 486 | + 0xb3f00000, | |
| 504 | 487 | 0x040c9850, |
| 505 | 488 | 0xbb0fc4b6, |
| 506 | 489 | 0x0c9800bc, |
| 507 | − 0x030d9802, | |
| 508 | − 0xf1080f98, | |
| 509 | − 0xf50200e7, | |
| 510 | − 0xf5016f21, | |
| 511 | − 0xf4025e21, | |
| 512 | − 0x12f40601, | |
| 513 | −/* 0x06a9: ctx_xfer_post */ | |
| 514 | − 0x7f21f507, | |
| 515 | −/* 0x06ad: ctx_xfer_done */ | |
| 516 | − 0xb421f502, | |
| 517 | − 0x0000f805, | |
| 518 | − 0x00000000, | |
| 490 | + 0x010d9800, | |
| 491 | + 0xf500e7f0, | |
| 492 | + 0xf0016f21, | |
| 493 | + 0xb7f101ac, | |
| 494 | + 0xb3f04000, | |
| 495 | + 0x040c9850, | |
| 496 | + 0xbb0fc4b6, | |
| 497 | + 0x0c9800bc, | |
| 498 | + 0x020d9801, | |
| 499 | + 0xf1060f98, | |
| 500 | + 0xf50800e7, | |
| 501 | + 0xf0016f21, | |
| 502 | + 0xa5f001ac, | |
| 503 | + 0x00b7f104, | |
| 504 | + 0x50b3f030, | |
| 505 | + 0xb6040c98, | |
| 506 | + 0xbcbb0fc4, | |
| 507 | + 0x020c9800, | |
| 508 | + 0x98030d98, | |
| 509 | + 0xe7f1080f, | |
| 510 | + 0x21f50200, | |
| 511 | + 0x21f5016f, | |
| 512 | + 0x01f4025e, | |
| 513 | + 0x0712f406, | |
| 514 | +/* 0x06ac: ctx_xfer_post */ | |
| 515 | + 0x027f21f5, | |
| 516 | +/* 0x06b0: ctx_xfer_done */ | |
| 517 | + 0x05b721f5, | |
| 518 | + 0x000000f8, | |
| 519 | 519 | 0x00000000, |
| 520 | 520 | 0x00000000, |
| 521 | 521 | 0x00000000, |
| @@ -276,7 +276,7 @@ uint32_t gk208_grgpc_code[] = { · 275 unmodified lines | ||
| 276 | 276 | 0x02020014, |
| 277 | 277 | 0xf6120040, |
| 278 | 278 | 0x04bd0002, |
| 279 | − 0xfe048141, | |
| 279 | + 0xfe048441, | |
| 280 | 280 | 0x00400010, |
| 281 | 281 | 0x0000f607, |
| 282 | 282 | 0x040204bd, |
| @@ -295,165 +295,165 @@ uint32_t gk208_grgpc_code[] = { · 12 unmodified lines | ||
| 295 | 295 | 0x01c90080, |
| 296 | 296 | 0xbd0002f6, |
| 297 | 297 | 0x0c308e04, |
| 298 | − 0xbd24bd50, | |
| 299 | −/* 0x0383: init_unk_loop */ | |
| 300 | − 0x7e44bd34, | |
| 301 | − 0xb0000065, | |
| 302 | − 0x0bf400f6, | |
| 303 | − 0xbb010f0e, | |
| 304 | − 0x4ffd04f2, | |
| 305 | − 0x0130b605, | |
| 306 | −/* 0x0398: init_unk_next */ | |
| 307 | − 0xb60120b6, | |
| 308 | − 0x26b004e0, | |
| 309 | − 0xe21bf401, | |
| 310 | −/* 0x03a4: init_unk_done */ | |
| 311 | − 0xb50703b5, | |
| 312 | − 0x00820804, | |
| 313 | − 0x22cf0201, | |
| 314 | − 0x9534bd00, | |
| 315 | − 0x00800825, | |
| 316 | − 0x05f601c0, | |
| 317 | − 0x8004bd00, | |
| 318 | − 0xf601c100, | |
| 319 | − 0x04bd0005, | |
| 320 | − 0x98000e98, | |
| 321 | − 0x207e010f, | |
| 322 | − 0x2fbb0001, | |
| 323 | − 0x003fbb00, | |
| 324 | − 0x98010e98, | |
| 325 | − 0x207e020f, | |
| 326 | − 0x0e980001, | |
| 327 | − 0x00effd05, | |
| 328 | − 0xbb002ebb, | |
| 329 | − 0x0e98003e, | |
| 330 | − 0x030f9802, | |
| 331 | − 0x0001207e, | |
| 332 | − 0xfd070e98, | |
| 333 | − 0x2ebb00ef, | |
| 334 | − 0x003ebb00, | |
| 335 | − 0x800235b6, | |
| 336 | − 0xf601d300, | |
| 337 | − 0x04bd0003, | |
| 338 | − 0xb60825b6, | |
| 339 | − 0x20b60635, | |
| 340 | − 0x0130b601, | |
| 341 | − 0xb60824b6, | |
| 342 | − 0x2fb20834, | |
| 343 | − 0x0002687e, | |
| 344 | − 0xbb002fbb, | |
| 345 | − 0x0080003f, | |
| 346 | − 0x03f60201, | |
| 347 | − 0xbd04bd00, | |
| 348 | − 0x1f29f024, | |
| 349 | − 0x02300080, | |
| 350 | − 0xbd0002f6, | |
| 351 | −/* 0x0445: main */ | |
| 352 | − 0x0031f404, | |
| 353 | − 0x0d0028f4, | |
| 354 | − 0x00377e24, | |
| 355 | − 0xf401f400, | |
| 356 | − 0xf404e4b0, | |
| 357 | − 0x81fe1d18, | |
| 358 | − 0xbd060201, | |
| 359 | − 0x0412fd20, | |
| 360 | − 0xfd01e4b6, | |
| 361 | − 0x18fe051e, | |
| 362 | − 0x05187e00, | |
| 363 | − 0xd40ef400, | |
| 364 | −/* 0x0474: main_not_ctx_xfer */ | |
| 365 | − 0xf010ef94, | |
| 366 | − 0xf87e01f5, | |
| 367 | − 0x0ef40002, | |
| 368 | −/* 0x0481: ih */ | |
| 369 | − 0xfe80f9c7, | |
| 370 | − 0x80f90188, | |
| 371 | − 0xa0f990f9, | |
| 372 | − 0xd0f9b0f9, | |
| 373 | − 0xf0f9e0f9, | |
| 374 | − 0x004a04bd, | |
| 375 | − 0x00aacf02, | |
| 376 | − 0xf404abc4, | |
| 377 | − 0x240d1f0b, | |
| 378 | − 0xcf1a004e, | |
| 379 | − 0x004f00ee, | |
| 380 | − 0x00ffcf19, | |
| 381 | − 0x0000047e, | |
| 382 | − 0x0040010e, | |
| 383 | − 0x000ef61d, | |
| 384 | −/* 0x04be: ih_no_fifo */ | |
| 385 | − 0x004004bd, | |
| 386 | − 0x000af601, | |
| 387 | − 0xf0fc04bd, | |
| 388 | − 0xd0fce0fc, | |
| 389 | − 0xa0fcb0fc, | |
| 390 | − 0x80fc90fc, | |
| 391 | − 0xfc0088fe, | |
| 392 | − 0x0032f480, | |
| 393 | −/* 0x04de: hub_barrier_done */ | |
| 394 | − 0x010f01f8, | |
| 395 | − 0xbb040e98, | |
| 396 | − 0xffb204fe, | |
| 397 | − 0x4094188e, | |
| 398 | − 0x00008f7e, | |
| 399 | −/* 0x04f2: ctx_redswitch */ | |
| 400 | − 0x200f00f8, | |
| 298 | + 0x01e5f050, | |
| 299 | + 0x34bd24bd, | |
| 300 | +/* 0x0386: init_unk_loop */ | |
| 301 | + 0x657e44bd, | |
| 302 | + 0xf6b00000, | |
| 303 | + 0x0e0bf400, | |
| 304 | + 0xf2bb010f, | |
| 305 | + 0x054ffd04, | |
| 306 | +/* 0x039b: init_unk_next */ | |
| 307 | + 0xb60130b6, | |
| 308 | + 0xe0b60120, | |
| 309 | + 0x0126b004, | |
| 310 | +/* 0x03a7: init_unk_done */ | |
| 311 | + 0xb5e21bf4, | |
| 312 | + 0x04b50703, | |
| 313 | + 0x01008208, | |
| 314 | + 0x0022cf02, | |
| 315 | + 0x259534bd, | |
| 316 | + 0xc0008008, | |
| 317 | + 0x0005f601, | |
| 318 | + 0x008004bd, | |
| 319 | + 0x05f601c1, | |
| 320 | + 0x9804bd00, | |
| 321 | + 0x0f98000e, | |
| 322 | + 0x01207e01, | |
| 323 | + 0x002fbb00, | |
| 324 | + 0x98003fbb, | |
| 325 | + 0x0f98010e, | |
| 326 | + 0x01207e02, | |
| 327 | + 0x050e9800, | |
| 328 | + 0xbb00effd, | |
| 329 | + 0x3ebb002e, | |
| 330 | + 0x020e9800, | |
| 331 | + 0x7e030f98, | |
| 332 | + 0x98000120, | |
| 333 | + 0xeffd070e, | |
| 334 | + 0x002ebb00, | |
| 335 | + 0xb6003ebb, | |
| 336 | + 0x00800235, | |
| 337 | + 0x03f601d3, | |
| 338 | + 0xb604bd00, | |
| 339 | + 0x35b60825, | |
| 340 | + 0x0120b606, | |
| 341 | + 0xb60130b6, | |
| 342 | + 0x34b60824, | |
| 343 | + 0x7e2fb208, | |
| 344 | + 0xbb000268, | |
| 345 | + 0x3fbb002f, | |
| 346 | + 0x01008000, | |
| 347 | + 0x0003f602, | |
| 348 | + 0x24bd04bd, | |
| 349 | + 0x801f29f0, | |
| 350 | + 0xf6023000, | |
| 351 | + 0x04bd0002, | |
| 352 | +/* 0x0448: main */ | |
| 353 | + 0xf40031f4, | |
| 354 | + 0x240d0028, | |
| 355 | + 0x0000377e, | |
| 356 | + 0xb0f401f4, | |
| 357 | + 0x18f404e4, | |
| 358 | + 0x0181fe1d, | |
| 359 | + 0x20bd0602, | |
| 360 | + 0xb60412fd, | |
| 361 | + 0x1efd01e4, | |
| 362 | + 0x0018fe05, | |
| 363 | + 0x00051b7e, | |
| 364 | +/* 0x0477: main_not_ctx_xfer */ | |
| 365 | + 0x94d40ef4, | |
| 366 | + 0xf5f010ef, | |
| 367 | + 0x02f87e01, | |
| 368 | + 0xc70ef400, | |
| 369 | +/* 0x0484: ih */ | |
| 370 | + 0x88fe80f9, | |
| 371 | + 0xf980f901, | |
| 372 | + 0xf9a0f990, | |
| 373 | + 0xf9d0f9b0, | |
| 374 | + 0xbdf0f9e0, | |
| 375 | + 0x02004a04, | |
| 376 | + 0xc400aacf, | |
| 377 | + 0x0bf404ab, | |
| 378 | + 0x4e240d1f, | |
| 379 | + 0xeecf1a00, | |
| 380 | + 0x19004f00, | |
| 381 | + 0x7e00ffcf, | |
| 382 | + 0x0e000004, | |
| 383 | + 0x1d004001, | |
| 384 | + 0xbd000ef6, | |
| 385 | +/* 0x04c1: ih_no_fifo */ | |
| 386 | + 0x01004004, | |
| 387 | + 0xbd000af6, | |
| 388 | + 0xfcf0fc04, | |
| 389 | + 0xfcd0fce0, | |
| 390 | + 0xfca0fcb0, | |
| 391 | + 0xfe80fc90, | |
| 392 | + 0x80fc0088, | |
| 393 | + 0xf80032f4, | |
| 394 | +/* 0x04e1: hub_barrier_done */ | |
| 395 | + 0x98010f01, | |
| 396 | + 0xfebb040e, | |
| 397 | + 0x8effb204, | |
| 398 | + 0x7e409418, | |
| 399 | + 0xf800008f, | |
| 400 | +/* 0x04f5: ctx_redswitch */ | |
| 401 | + 0x80200f00, | |
| 402 | + 0xf6018500, | |
| 403 | + 0x04bd000f, | |
| 404 | +/* 0x0502: ctx_redswitch_delay */ | |
| 405 | + 0xe2b6080e, | |
| 406 | + 0xfd1bf401, | |
| 407 | + 0x0800f5f1, | |
| 408 | + 0x0200f5f1, | |
| 401 | 409 | 0x01850080, |
| 402 | 410 | 0xbd000ff6, |
| 403 | −/* 0x04ff: ctx_redswitch_delay */ | |
| 404 | − 0xb6080e04, | |
| 405 | − 0x1bf401e2, | |
| 406 | − 0x00f5f1fd, | |
| 407 | − 0x00f5f108, | |
| 408 | − 0x85008002, | |
| 409 | − 0x000ff601, | |
| 410 | − 0x00f804bd, | |
| 411 | −/* 0x0518: ctx_xfer */ | |
| 412 | − 0x02810080, | |
| 413 | − 0xbd000ff6, | |
| 414 | − 0x0711f404, | |
| 415 | − 0x0004f27e, | |
| 416 | −/* 0x0528: ctx_xfer_not_load */ | |
| 417 | − 0x0002167e, | |
| 418 | − 0xfc8024bd, | |
| 419 | − 0x02f60247, | |
| 420 | − 0xf004bd00, | |
| 421 | − 0x20b6012c, | |
| 422 | − 0x4afc8003, | |
| 411 | +/* 0x051b: ctx_xfer */ | |
| 412 | + 0x8000f804, | |
| 413 | + 0xf6028100, | |
| 414 | + 0x04bd000f, | |
| 415 | + 0x7e0711f4, | |
| 416 | +/* 0x052b: ctx_xfer_not_load */ | |
| 417 | + 0x7e0004f5, | |
| 418 | + 0xbd000216, | |
| 419 | + 0x47fc8024, | |
| 423 | 420 | 0x0002f602, |
| 424 | − 0xacf004bd, | |
| 425 | − 0x02a5f001, | |
| 426 | − 0x5000008b, | |
| 427 | − 0xb6040c98, | |
| 428 | − 0xbcbb0fc4, | |
| 429 | − 0x000c9800, | |
| 430 | − 0x0e010d98, | |
| 431 | − 0x013d7e00, | |
| 432 | − 0x01acf000, | |
| 433 | − 0x5040008b, | |
| 434 | − 0xb6040c98, | |
| 435 | − 0xbcbb0fc4, | |
| 436 | − 0x010c9800, | |
| 437 | − 0x98020d98, | |
| 438 | − 0x004e060f, | |
| 439 | − 0x013d7e08, | |
| 440 | − 0x01acf000, | |
| 441 | − 0x8b04a5f0, | |
| 442 | − 0x98503000, | |
| 421 | + 0x2cf004bd, | |
| 422 | + 0x0320b601, | |
| 423 | + 0x024afc80, | |
| 424 | + 0xbd0002f6, | |
| 425 | + 0x01acf004, | |
| 426 | + 0x8b02a5f0, | |
| 427 | + 0x98500000, | |
| 443 | 428 | 0xc4b6040c, |
| 444 | 429 | 0x00bcbb0f, |
| 445 | − 0x98020c98, | |
| 446 | − 0x0f98030d, | |
| 447 | − 0x02004e08, | |
| 430 | + 0x98000c98, | |
| 431 | + 0x000e010d, | |
| 448 | 432 | 0x00013d7e, |
| 449 | − 0x00020a7e, | |
| 450 | − 0xf40601f4, | |
| 451 | −/* 0x05b2: ctx_xfer_post */ | |
| 452 | − 0x277e0712, | |
| 453 | −/* 0x05b6: ctx_xfer_done */ | |
| 454 | − 0xde7e0002, | |
| 455 | − 0x00f80004, | |
| 456 | − 0x00000000, | |
| 433 | + 0x8b01acf0, | |
| 434 | + 0x98504000, | |
| 435 | + 0xc4b6040c, | |
| 436 | + 0x00bcbb0f, | |
| 437 | + 0x98010c98, | |
| 438 | + 0x0f98020d, | |
| 439 | + 0x08004e06, | |
| 440 | + 0x00013d7e, | |
| 441 | + 0xf001acf0, | |
| 442 | + 0x008b04a5, | |
| 443 | + 0x0c985030, | |
| 444 | + 0x0fc4b604, | |
| 445 | + 0x9800bcbb, | |
| 446 | + 0x0d98020c, | |
| 447 | + 0x080f9803, | |
| 448 | + 0x7e02004e, | |
| 449 | + 0x7e00013d, | |
| 450 | + 0xf400020a, | |
| 451 | + 0x12f40601, | |
| 452 | +/* 0x05b5: ctx_xfer_post */ | |
| 453 | + 0x02277e07, | |
| 454 | +/* 0x05b9: ctx_xfer_done */ | |
| 455 | + 0x04e17e00, | |
| 456 | + 0x0000f800, | |
| 457 | 457 | 0x00000000, |
| 458 | 458 | 0x00000000, |
| 459 | 459 | 0x00000000, |
| @@ -289,7 +289,7 @@ uint32_t gm107_grgpc_code[] = { · 288 unmodified lines | ||
| 289 | 289 | 0x020014fe, |
| 290 | 290 | 0x12004002, |
| 291 | 291 | 0xbd0002f6, |
| 292 | − 0x05b04104, | |
| 292 | + 0x05b34104, | |
| 293 | 293 | 0x400010fe, |
| 294 | 294 | 0x00f60700, |
| 295 | 295 | 0x0204bd00, |
| @@ -308,259 +308,259 @@ uint32_t gm107_grgpc_code[] = { · 12 unmodified lines | ||
| 308 | 308 | 0xc900800f, |
| 309 | 309 | 0x0002f601, |
| 310 | 310 | 0x308e04bd, |
| 311 | − 0x24bd500c, | |
| 312 | − 0x44bd34bd, | |
| 313 | −/* 0x03b0: init_unk_loop */ | |
| 314 | − 0x0000657e, | |
| 315 | − 0xf400f6b0, | |
| 316 | − 0x010f0e0b, | |
| 317 | − 0xfd04f2bb, | |
| 318 | − 0x30b6054f, | |
| 319 | −/* 0x03c5: init_unk_next */ | |
| 320 | − 0x0120b601, | |
| 321 | − 0xb004e0b6, | |
| 322 | − 0x1bf40226, | |
| 323 | −/* 0x03d1: init_unk_done */ | |
| 324 | − 0x0703b5e2, | |
| 325 | − 0x820804b5, | |
| 326 | − 0xcf020100, | |
| 327 | − 0x34bd0022, | |
| 328 | − 0x80082595, | |
| 329 | − 0xf601c000, | |
| 311 | + 0xe5f0500c, | |
| 312 | + 0xbd24bd01, | |
| 313 | +/* 0x03b3: init_unk_loop */ | |
| 314 | + 0x7e44bd34, | |
| 315 | + 0xb0000065, | |
| 316 | + 0x0bf400f6, | |
| 317 | + 0xbb010f0e, | |
| 318 | + 0x4ffd04f2, | |
| 319 | + 0x0130b605, | |
| 320 | +/* 0x03c8: init_unk_next */ | |
| 321 | + 0xb60120b6, | |
| 322 | + 0x26b004e0, | |
| 323 | + 0xe21bf402, | |
| 324 | +/* 0x03d4: init_unk_done */ | |
| 325 | + 0xb50703b5, | |
| 326 | + 0x00820804, | |
| 327 | + 0x22cf0201, | |
| 328 | + 0x9534bd00, | |
| 329 | + 0x00800825, | |
| 330 | + 0x05f601c0, | |
| 331 | + 0x8004bd00, | |
| 332 | + 0xf601c100, | |
| 330 | 333 | 0x04bd0005, |
| 331 | − 0x01c10080, | |
| 332 | − 0xbd0005f6, | |
| 333 | − 0x000e9804, | |
| 334 | − 0x7e010f98, | |
| 335 | − 0xbb000120, | |
| 336 | − 0x3fbb002f, | |
| 337 | − 0x010e9800, | |
| 338 | − 0x7e020f98, | |
| 339 | − 0x98000120, | |
| 340 | − 0xeffd050e, | |
| 341 | − 0x002ebb00, | |
| 342 | − 0x98003ebb, | |
| 343 | − 0x0f98020e, | |
| 344 | − 0x01207e03, | |
| 345 | − 0x070e9800, | |
| 346 | − 0xbb00effd, | |
| 347 | − 0x3ebb002e, | |
| 348 | − 0x0235b600, | |
| 349 | − 0x01d30080, | |
| 350 | − 0xbd0003f6, | |
| 351 | − 0x0825b604, | |
| 352 | − 0xb60635b6, | |
| 353 | − 0x30b60120, | |
| 354 | − 0x0824b601, | |
| 355 | − 0xb20834b6, | |
| 356 | − 0x02687e2f, | |
| 357 | − 0x002fbb00, | |
| 358 | − 0x0f003fbb, | |
| 359 | − 0x8effb23f, | |
| 360 | − 0xf0501d60, | |
| 361 | − 0x8f7e01e5, | |
| 362 | − 0x0c0f0000, | |
| 363 | − 0xa88effb2, | |
| 364 | − 0xe5f0501d, | |
| 365 | − 0x008f7e01, | |
| 366 | − 0x03147e00, | |
| 367 | − 0xb23f0f00, | |
| 368 | − 0x1d608eff, | |
| 369 | − 0x01e5f050, | |
| 370 | − 0x00008f7e, | |
| 371 | − 0xffb2000f, | |
| 372 | − 0x501d9c8e, | |
| 373 | − 0x7e01e5f0, | |
| 374 | − 0x0f00008f, | |
| 375 | − 0x03147e01, | |
| 376 | − 0x8effb200, | |
| 334 | + 0x98000e98, | |
| 335 | + 0x207e010f, | |
| 336 | + 0x2fbb0001, | |
| 337 | + 0x003fbb00, | |
| 338 | + 0x98010e98, | |
| 339 | + 0x207e020f, | |
| 340 | + 0x0e980001, | |
| 341 | + 0x00effd05, | |
| 342 | + 0xbb002ebb, | |
| 343 | + 0x0e98003e, | |
| 344 | + 0x030f9802, | |
| 345 | + 0x0001207e, | |
| 346 | + 0xfd070e98, | |
| 347 | + 0x2ebb00ef, | |
| 348 | + 0x003ebb00, | |
| 349 | + 0x800235b6, | |
| 350 | + 0xf601d300, | |
| 351 | + 0x04bd0003, | |
| 352 | + 0xb60825b6, | |
| 353 | + 0x20b60635, | |
| 354 | + 0x0130b601, | |
| 355 | + 0xb60824b6, | |
| 356 | + 0x2fb20834, | |
| 357 | + 0x0002687e, | |
| 358 | + 0xbb002fbb, | |
| 359 | + 0x3f0f003f, | |
| 360 | + 0x501d608e, | |
| 361 | + 0xb201e5f0, | |
| 362 | + 0x008f7eff, | |
| 363 | + 0x8e0c0f00, | |
| 377 | 364 | 0xf0501da8, |
| 378 | − 0x8f7e01e5, | |
| 379 | − 0xff0f0000, | |
| 380 | − 0x988effb2, | |
| 365 | + 0xffb201e5, | |
| 366 | + 0x00008f7e, | |
| 367 | + 0x0003147e, | |
| 368 | + 0x608e3f0f, | |
| 381 | 369 | 0xe5f0501d, |
| 382 | − 0x008f7e01, | |
| 383 | − 0xb2020f00, | |
| 384 | − 0x1da88eff, | |
| 370 | + 0x7effb201, | |
| 371 | + 0x0f00008f, | |
| 372 | + 0x1d9c8e00, | |
| 385 | 373 | 0x01e5f050, |
| 386 | − 0x00008f7e, | |
| 374 | + 0x8f7effb2, | |
| 375 | + 0x010f0000, | |
| 387 | 376 | 0x0003147e, |
| 388 | − 0x85050498, | |
| 389 | − 0x98504000, | |
| 390 | − 0x64b60406, | |
| 391 | − 0x0056bb0f, | |
| 392 | −/* 0x04e0: tpc_strand_init_tpc_loop */ | |
| 393 | − 0x05705eb8, | |
| 394 | − 0x00657e00, | |
| 395 | − 0xbdf6b200, | |
| 396 | −/* 0x04ed: tpc_strand_init_idx_loop */ | |
| 397 | − 0x605eb874, | |
| 398 | − 0x7fb20005, | |
| 399 | − 0x00008f7e, | |
| 400 | − 0x05885eb8, | |
| 401 | − 0x082f9500, | |
| 402 | − 0x00008f7e, | |
| 403 | − 0x058c5eb8, | |
| 404 | − 0x082f9500, | |
| 377 | + 0x501da88e, | |
| 378 | + 0xb201e5f0, | |
| 379 | + 0x008f7eff, | |
| 380 | + 0x8eff0f00, | |
| 381 | + 0xf0501d98, | |
| 382 | + 0xffb201e5, | |
| 405 | 383 | 0x00008f7e, |
| 406 | − 0x05905eb8, | |
| 407 | − 0x00657e00, | |
| 408 | − 0x06f5b600, | |
| 409 | − 0xb601f0b6, | |
| 410 | − 0x2fbb08f4, | |
| 411 | − 0x003fbb00, | |
| 412 | − 0xb60170b6, | |
| 413 | − 0x1bf40162, | |
| 414 | − 0x0050b7bf, | |
| 415 | − 0x0142b608, | |
| 416 | − 0x0fa81bf4, | |
| 417 | − 0x8effb23f, | |
| 418 | − 0xf0501d60, | |
| 419 | − 0x8f7e01e5, | |
| 420 | − 0x0d0f0000, | |
| 421 | − 0xa88effb2, | |
| 384 | + 0xa88e020f, | |
| 422 | 385 | 0xe5f0501d, |
| 423 | − 0x008f7e01, | |
| 424 | − 0x03147e00, | |
| 425 | − 0x01008000, | |
| 426 | − 0x0003f602, | |
| 427 | − 0x24bd04bd, | |
| 428 | − 0x801f29f0, | |
| 429 | − 0xf6023000, | |
| 430 | − 0x04bd0002, | |
| 431 | −/* 0x0574: main */ | |
| 432 | − 0xf40031f4, | |
| 433 | − 0x240d0028, | |
| 434 | − 0x0000377e, | |
| 435 | − 0xb0f401f4, | |
| 436 | − 0x18f404e4, | |
| 437 | − 0x0181fe1d, | |
| 438 | − 0x20bd0602, | |
| 439 | − 0xb60412fd, | |
| 440 | − 0x1efd01e4, | |
| 441 | − 0x0018fe05, | |
| 442 | − 0x0006477e, | |
| 443 | −/* 0x05a3: main_not_ctx_xfer */ | |
| 444 | − 0x94d40ef4, | |
| 445 | − 0xf5f010ef, | |
| 446 | − 0x02f87e01, | |
| 447 | − 0xc70ef400, | |
| 448 | −/* 0x05b0: ih */ | |
| 449 | − 0x88fe80f9, | |
| 450 | − 0xf980f901, | |
| 451 | − 0xf9a0f990, | |
| 452 | − 0xf9d0f9b0, | |
| 453 | − 0xbdf0f9e0, | |
| 454 | − 0x02004a04, | |
| 455 | − 0xc400aacf, | |
| 456 | − 0x0bf404ab, | |
| 457 | − 0x4e240d1f, | |
| 458 | − 0xeecf1a00, | |
| 459 | − 0x19004f00, | |
| 460 | − 0x7e00ffcf, | |
| 461 | − 0x0e000004, | |
| 462 | − 0x1d004001, | |
| 463 | − 0xbd000ef6, | |
| 464 | −/* 0x05ed: ih_no_fifo */ | |
| 465 | − 0x01004004, | |
| 466 | − 0xbd000af6, | |
| 467 | − 0xfcf0fc04, | |
| 468 | − 0xfcd0fce0, | |
| 469 | − 0xfca0fcb0, | |
| 470 | − 0xfe80fc90, | |
| 471 | − 0x80fc0088, | |
| 472 | − 0xf80032f4, | |
| 473 | −/* 0x060d: hub_barrier_done */ | |
| 474 | − 0x98010f01, | |
| 475 | − 0xfebb040e, | |
| 476 | − 0x8effb204, | |
| 477 | − 0x7e409418, | |
| 478 | − 0xf800008f, | |
| 479 | −/* 0x0621: ctx_redswitch */ | |
| 480 | − 0x80200f00, | |
| 386 | + 0x7effb201, | |
| 387 | + 0x7e00008f, | |
| 388 | + 0x98000314, | |
| 389 | + 0x00850504, | |
| 390 | + 0x06985040, | |
| 391 | + 0x0f64b604, | |
| 392 | +/* 0x04e3: tpc_strand_init_tpc_loop */ | |
| 393 | + 0xb80056bb, | |
| 394 | + 0x0005705e, | |
| 395 | + 0x0000657e, | |
| 396 | + 0x74bdf6b2, | |
| 397 | +/* 0x04f0: tpc_strand_init_idx_loop */ | |
| 398 | + 0x05605eb8, | |
| 399 | + 0x7e7fb200, | |
| 400 | + 0xb800008f, | |
| 401 | + 0x0005885e, | |
| 402 | + 0x7e082f95, | |
| 403 | + 0xb800008f, | |
| 404 | + 0x00058c5e, | |
| 405 | + 0x7e082f95, | |
| 406 | + 0xb800008f, | |
| 407 | + 0x0005905e, | |
| 408 | + 0x0000657e, | |
| 409 | + 0xb606f5b6, | |
| 410 | + 0xf4b601f0, | |
| 411 | + 0x002fbb08, | |
| 412 | + 0xb6003fbb, | |
| 413 | + 0x62b60170, | |
| 414 | + 0xbf1bf401, | |
| 415 | + 0x080050b7, | |
| 416 | + 0xf40142b6, | |
| 417 | + 0x3f0fa81b, | |
| 418 | + 0x501d608e, | |
| 419 | + 0xb201e5f0, | |
| 420 | + 0x008f7eff, | |
| 421 | + 0x8e0d0f00, | |
| 422 | + 0xf0501da8, | |
| 423 | + 0xffb201e5, | |
| 424 | + 0x00008f7e, | |
| 425 | + 0x0003147e, | |
| 426 | + 0x02010080, | |
| 427 | + 0xbd0003f6, | |
| 428 | + 0xf024bd04, | |
| 429 | + 0x00801f29, | |
| 430 | + 0x02f60230, | |
| 431 | +/* 0x0577: main */ | |
| 432 | + 0xf404bd00, | |
| 433 | + 0x28f40031, | |
| 434 | + 0x7e240d00, | |
| 435 | + 0xf4000037, | |
| 436 | + 0xe4b0f401, | |
| 437 | + 0x1d18f404, | |
| 438 | + 0x020181fe, | |
| 439 | + 0xfd20bd06, | |
| 440 | + 0xe4b60412, | |
| 441 | + 0x051efd01, | |
| 442 | + 0x7e0018fe, | |
| 443 | + 0xf400064a, | |
| 444 | +/* 0x05a6: main_not_ctx_xfer */ | |
| 445 | + 0xef94d40e, | |
| 446 | + 0x01f5f010, | |
| 447 | + 0x0002f87e, | |
| 448 | +/* 0x05b3: ih */ | |
| 449 | + 0xf9c70ef4, | |
| 450 | + 0x0188fe80, | |
| 451 | + 0x90f980f9, | |
| 452 | + 0xb0f9a0f9, | |
| 453 | + 0xe0f9d0f9, | |
| 454 | + 0x04bdf0f9, | |
| 455 | + 0xcf02004a, | |
| 456 | + 0xabc400aa, | |
| 457 | + 0x1f0bf404, | |
| 458 | + 0x004e240d, | |
| 459 | + 0x00eecf1a, | |
| 460 | + 0xcf19004f, | |
| 461 | + 0x047e00ff, | |
| 462 | + 0x010e0000, | |
| 463 | + 0xf61d0040, | |
| 464 | + 0x04bd000e, | |
| 465 | +/* 0x05f0: ih_no_fifo */ | |
| 466 | + 0xf6010040, | |
| 467 | + 0x04bd000a, | |
| 468 | + 0xe0fcf0fc, | |
| 469 | + 0xb0fcd0fc, | |
| 470 | + 0x90fca0fc, | |
| 471 | + 0x88fe80fc, | |
| 472 | + 0xf480fc00, | |
| 473 | + 0x01f80032, | |
| 474 | +/* 0x0610: hub_barrier_done */ | |
| 475 | + 0x0e98010f, | |
| 476 | + 0x04febb04, | |
| 477 | + 0x188effb2, | |
| 478 | + 0x8f7e4094, | |
| 479 | + 0x00f80000, | |
| 480 | +/* 0x0624: ctx_redswitch */ | |
| 481 | + 0x0080200f, | |
| 482 | + 0x0ff60185, | |
| 483 | + 0x0e04bd00, | |
| 484 | +/* 0x0631: ctx_redswitch_delay */ | |
| 485 | + 0x01e2b608, | |
| 486 | + 0xf1fd1bf4, | |
| 487 | + 0xf10800f5, | |
| 488 | + 0x800200f5, | |
| 481 | 489 | 0xf6018500, |
| 482 | 490 | 0x04bd000f, |
| 483 | −/* 0x062e: ctx_redswitch_delay */ | |
| 484 | − 0xe2b6080e, | |
| 485 | − 0xfd1bf401, | |
| 486 | − 0x0800f5f1, | |
| 487 | − 0x0200f5f1, | |
| 488 | − 0x01850080, | |
| 489 | − 0xbd000ff6, | |
| 490 | −/* 0x0647: ctx_xfer */ | |
| 491 | − 0x8000f804, | |
| 492 | − 0xf6028100, | |
| 493 | − 0x04bd000f, | |
| 494 | − 0xc48effb2, | |
| 495 | − 0xe5f0501d, | |
| 496 | − 0x008f7e01, | |
| 497 | − 0x0711f400, | |
| 498 | − 0x0006217e, | |
| 499 | −/* 0x0664: ctx_xfer_not_load */ | |
| 500 | − 0x0002167e, | |
| 501 | − 0xfc8024bd, | |
| 502 | − 0x02f60247, | |
| 503 | − 0xf004bd00, | |
| 504 | − 0x20b6012c, | |
| 505 | − 0x4afc8003, | |
| 491 | +/* 0x064a: ctx_xfer */ | |
| 492 | + 0x008000f8, | |
| 493 | + 0x0ff60281, | |
| 494 | + 0x8e04bd00, | |
| 495 | + 0xf0501dc4, | |
| 496 | + 0xffb201e5, | |
| 497 | + 0x00008f7e, | |
| 498 | + 0x7e0711f4, | |
| 499 | +/* 0x0667: ctx_xfer_not_load */ | |
| 500 | + 0x7e000624, | |
| 501 | + 0xbd000216, | |
| 502 | + 0x47fc8024, | |
| 506 | 503 | 0x0002f602, |
| 507 | − 0x0c0f04bd, | |
| 508 | − 0xa88effb2, | |
| 509 | − 0xe5f0501d, | |
| 510 | − 0x008f7e01, | |
| 511 | − 0x03147e00, | |
| 512 | − 0xb23f0f00, | |
| 513 | − 0x1d608eff, | |
| 514 | − 0x01e5f050, | |
| 504 | + 0x2cf004bd, | |
| 505 | + 0x0320b601, | |
| 506 | + 0x024afc80, | |
| 507 | + 0xbd0002f6, | |
| 508 | + 0x8e0c0f04, | |
| 509 | + 0xf0501da8, | |
| 510 | + 0xffb201e5, | |
| 515 | 511 | 0x00008f7e, |
| 516 | − 0xffb2000f, | |
| 517 | − 0x501d9c8e, | |
| 518 | − 0x7e01e5f0, | |
| 512 | + 0x0003147e, | |
| 513 | + 0x608e3f0f, | |
| 514 | + 0xe5f0501d, | |
| 515 | + 0x7effb201, | |
| 519 | 516 | 0x0f00008f, |
| 520 | − 0x03147e01, | |
| 521 | − 0x01fcf000, | |
| 522 | − 0xb203f0b6, | |
| 523 | − 0x1da88eff, | |
| 517 | + 0x1d9c8e00, | |
| 524 | 518 | 0x01e5f050, |
| 525 | − 0x00008f7e, | |
| 526 | − 0xf001acf0, | |
| 527 | − 0x008b02a5, | |
| 528 | − 0x0c985000, | |
| 529 | − 0x0fc4b604, | |
| 530 | − 0x9800bcbb, | |
| 531 | − 0x0d98000c, | |
| 532 | − 0x7e000e01, | |
| 533 | − 0xf000013d, | |
| 534 | − 0x008b01ac, | |
| 535 | − 0x0c985040, | |
| 536 | − 0x0fc4b604, | |
| 537 | − 0x9800bcbb, | |
| 538 | − 0x0d98010c, | |
| 539 | − 0x060f9802, | |
| 540 | − 0x7e08004e, | |
| 541 | − 0xf000013d, | |
| 519 | + 0x8f7effb2, | |
| 520 | + 0x010f0000, | |
| 521 | + 0x0003147e, | |
| 522 | + 0xb601fcf0, | |
| 523 | + 0xa88e03f0, | |
| 524 | + 0xe5f0501d, | |
| 525 | + 0x7effb201, | |
| 526 | + 0xf000008f, | |
| 542 | 527 | 0xa5f001ac, |
| 543 | − 0x30008b04, | |
| 528 | + 0x00008b02, | |
| 544 | 529 | 0x040c9850, |
| 545 | 530 | 0xbb0fc4b6, |
| 546 | 531 | 0x0c9800bc, |
| 547 | − 0x030d9802, | |
| 548 | − 0x4e080f98, | |
| 549 | − 0x3d7e0200, | |
| 550 | − 0x0a7e0001, | |
| 551 | − 0x147e0002, | |
| 552 | − 0x01f40003, | |
| 553 | − 0x1a12f406, | |
| 554 | −/* 0x073c: ctx_xfer_post */ | |
| 555 | − 0x0002277e, | |
| 556 | − 0xffb20d0f, | |
| 557 | − 0x501da88e, | |
| 558 | − 0x7e01e5f0, | |
| 559 | − 0x7e00008f, | |
| 560 | −/* 0x0753: ctx_xfer_done */ | |
| 561 | − 0x7e000314, | |
| 562 | − 0xf800060d, | |
| 563 | − 0x00000000, | |
| 532 | + 0x010d9800, | |
| 533 | + 0x3d7e000e, | |
| 534 | + 0xacf00001, | |
| 535 | + 0x40008b01, | |
| 536 | + 0x040c9850, | |
| 537 | + 0xbb0fc4b6, | |
| 538 | + 0x0c9800bc, | |
| 539 | + 0x020d9801, | |
| 540 | + 0x4e060f98, | |
| 541 | + 0x3d7e0800, | |
| 542 | + 0xacf00001, | |
| 543 | + 0x04a5f001, | |
| 544 | + 0x5030008b, | |
| 545 | + 0xb6040c98, | |
| 546 | + 0xbcbb0fc4, | |
| 547 | + 0x020c9800, | |
| 548 | + 0x98030d98, | |
| 549 | + 0x004e080f, | |
| 550 | + 0x013d7e02, | |
| 551 | + 0x020a7e00, | |
| 552 | + 0x03147e00, | |
| 553 | + 0x0601f400, | |
| 554 | +/* 0x073f: ctx_xfer_post */ | |
| 555 | + 0x7e1a12f4, | |
| 556 | + 0x0f000227, | |
| 557 | + 0x1da88e0d, | |
| 558 | + 0x01e5f050, | |
| 559 | + 0x8f7effb2, | |
| 560 | + 0x147e0000, | |
| 561 | +/* 0x0756: ctx_xfer_done */ | |
| 562 | + 0x107e0003, | |
| 563 | + 0x00f80006, | |
| 564 | 564 | 0x00000000, |
| 565 | 565 | 0x00000000, |
| 566 | 566 | 0x00000000, |
| @@ -143,7 +143,7 @@ gf100_gr_zbc_depth_get(struct gf100_gr *gr, int format, · 142 unmodified lines | ||
| 143 | 143 | static int |
| 144 | 144 | gf100_fermi_mthd_zbc_color(struct nvkm_object *object, void *data, u32 size) |
| 145 | 145 | { |
| 146 | − struct gf100_gr *gr = (void *)object->engine; | |
| 146 | + struct gf100_gr *gr = gf100_gr(nvkm_gr(object->engine)); | |
| 147 | 147 | union { |
| 148 | 148 | struct fermi_a_zbc_color_v0 v0; |
| 149 | 149 | } *args = data; |
| @@ -189,7 +189,7 @@ gf100_fermi_mthd_zbc_color(struct nvkm_object *object, void *data, u32 size) · 39 unmodified lines | ||
| 189 | 189 | static int |
| 190 | 190 | gf100_fermi_mthd_zbc_depth(struct nvkm_object *object, void *data, u32 size) |
| 191 | 191 | { |
| 192 | − struct gf100_gr *gr = (void *)object->engine; | |
| 192 | + struct gf100_gr *gr = gf100_gr(nvkm_gr(object->engine)); | |
| 193 | 193 | union { |
| 194 | 194 | struct fermi_a_zbc_depth_v0 v0; |
| 195 | 195 | } *args = data; |
| @@ -1530,6 +1530,8 @@ gf100_gr_oneinit(struct nvkm_gr *base) · 1334 unmodified lines | ||
| 1530 | 1530 | gr->ppc_nr[i] = gr->func->ppc_nr; |
| 1531 | 1531 | for (j = 0; j < gr->ppc_nr[i]; j++) { |
| 1532 | 1532 | u8 mask = nvkm_rd32(device, GPC_UNIT(i, 0x0c30 + (j * 4))); |
| 1533 | + if (mask) | |
| 1534 | + gr->ppc_mask[i] |= (1 << j); | |
| 1533 | 1535 | gr->ppc_tpc_nr[i][j] = hweight8(mask); |
| 1534 | 1536 | } |
| 1535 | 1537 | } |
| @@ -97,6 +97,7 @@ struct gf100_gr { · 96 unmodified lines | ||
| 97 | 97 | u8 tpc_nr[GPC_MAX]; |
| 98 | 98 | u8 tpc_total; |
| 99 | 99 | u8 ppc_nr[GPC_MAX]; |
| 100 | + u8 ppc_mask[GPC_MAX]; | |
| 100 | 101 | u8 ppc_tpc_nr[GPC_MAX][4]; |
| 101 | 102 | |
| 102 | 103 | struct nvkm_memory *unk4188b4; |
| @@ -97,7 +97,9 @@ static void * · 96 unmodified lines | ||
| 97 | 97 | nvkm_instobj_dtor(struct nvkm_memory *memory) |
| 98 | 98 | { |
| 99 | 99 | struct nvkm_instobj *iobj = nvkm_instobj(memory); |
| 100 | + spin_lock(&iobj->imem->lock); | |
| 100 | 101 | list_del(&iobj->head); |
| 102 | + spin_unlock(&iobj->imem->lock); | |
| 101 | 103 | nvkm_memory_del(&iobj->parent); |
| 102 | 104 | return iobj; |
| 103 | 105 | } |
| @@ -190,7 +192,9 @@ nvkm_instobj_new(struct nvkm_instmem *imem, u32 size, u32 align, bool zero, · 86 unmodified lines | ||
| 190 | 192 | nvkm_memory_ctor(&nvkm_instobj_func_slow, &iobj->memory); |
| 191 | 193 | iobj->parent = memory; |
| 192 | 194 | iobj->imem = imem; |
| 195 | + spin_lock(&iobj->imem->lock); | |
| 193 | 196 | list_add_tail(&iobj->head, &imem->list); |
| 197 | + spin_unlock(&iobj->imem->lock); | |
| 194 | 198 | memory = &iobj->memory; |
| 195 | 199 | } |
| 196 | 200 | |
| @@ -309,5 +313,6 @@ nvkm_instmem_ctor(const struct nvkm_instmem_func *func, · 112 unmodified lines | ||
| 309 | 313 | { |
| 310 | 314 | nvkm_subdev_ctor(&nvkm_instmem, device, index, 0, &imem->subdev); |
| 311 | 315 | imem->func = func; |
| 316 | + spin_lock_init(&imem->lock); | |
| 312 | 317 | INIT_LIST_HEAD(&imem->list); |
| 313 | 318 | } |
| @@ -81,9 +81,7 @@ gk104_pmu_pgob(struct nvkm_pmu *pmu, bool enable) · 80 unmodified lines | ||
| 81 | 81 | nvkm_mask(device, 0x000200, 0x00001000, 0x00001000); |
| 82 | 82 | nvkm_rd32(device, 0x000200); |
| 83 | 83 | |
| 84 | − if ( nvkm_boolopt(device->cfgopt, "War00C800_0", | |
| 85 | − device->quirk ? device->quirk->War00C800_0 : false)) { | |
| 86 | − nvkm_info(&pmu->subdev, "hw bug workaround enabled\n"); | |
| 84 | + if (nvkm_boolopt(device->cfgopt, "War00C800_0", true)) { | |
| 87 | 85 | switch (device->chipset) { |
| 88 | 86 | case 0xe4: |
| 89 | 87 | magic(device, 0x04000000); |
| @@ -59,7 +59,7 @@ gk104_volt_set(struct nvkm_volt *base, u32 uv) · 58 unmodified lines | ||
| 59 | 59 | duty = (uv - bios->base) * div / bios->pwm_range; |
| 60 | 60 | |
| 61 | 61 | nvkm_wr32(device, 0x20340, div); |
| 62 | − nvkm_wr32(device, 0x20344, 0x8000000 | duty); | |
| 62 | + nvkm_wr32(device, 0x20344, 0x80000000 | duty); | |
| 63 | 63 | |
| 64 | 64 | return 0; |
| 65 | 65 | } |
| @@ -4173,11 +4173,7 @@ void cik_ring_ib_execute(struct radeon_device *rdev, struct radeon_ib *ib) · 4172 unmodified lines | ||
| 4173 | 4173 | control |= ib->length_dw | (vm_id << 24); |
| 4174 | 4174 | |
| 4175 | 4175 | radeon_ring_write(ring, header); |
| 4176 | − radeon_ring_write(ring, | |
| 4177 | −#ifdef __BIG_ENDIAN | |
| 4178 | − (2 << 0) | | |
| 4179 | −#endif | |
| 4180 | − (ib->gpu_addr & 0xFFFFFFFC)); | |
| 4176 | + radeon_ring_write(ring, (ib->gpu_addr & 0xFFFFFFFC)); | |
| 4181 | 4177 | radeon_ring_write(ring, upper_32_bits(ib->gpu_addr) & 0xFFFF); |
| 4182 | 4178 | radeon_ring_write(ring, control); |
| 4183 | 4179 | } |
| @@ -8472,7 +8468,7 @@ restart_ih: · 4288 unmodified lines | ||
| 8472 | 8468 | if (queue_dp) |
| 8473 | 8469 | schedule_work(&rdev->dp_work); |
| 8474 | 8470 | if (queue_hotplug) |
| 8475 | − schedule_work(&rdev->hotplug_work); | |
| 8471 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 8476 | 8472 | if (queue_reset) { |
| 8477 | 8473 | rdev->needs_reset = true; |
| 8478 | 8474 | wake_up_all(&rdev->fence_queue); |
| @@ -9630,6 +9626,9 @@ static void dce8_program_watermarks(struct radeon_device *rdev, · 1151 unmodified lines | ||
| 9630 | 9626 | (rdev->disp_priority == 2)) { |
| 9631 | 9627 | DRM_DEBUG_KMS("force priority to high\n"); |
| 9632 | 9628 | } |
| 9629 | + | |
| 9630 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 9631 | + radeon_crtc->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 9633 | 9632 | } |
| 9634 | 9633 | |
| 9635 | 9634 | /* select wm A */ |
| @@ -2372,6 +2372,9 @@ static void evergreen_program_watermarks(struct radeon_device *rdev, · 2371 unmodified lines | ||
| 2372 | 2372 | c.full = dfixed_div(c, a); |
| 2373 | 2373 | priority_b_mark = dfixed_trunc(c); |
| 2374 | 2374 | priority_b_cnt |= priority_b_mark & PRIORITY_MARK_MASK; |
| 2375 | + | |
| 2376 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 2377 | + radeon_crtc->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 2375 | 2378 | } |
| 2376 | 2379 | |
| 2377 | 2380 | /* select wm A */ |
| @@ -5344,7 +5347,7 @@ restart_ih: · 2966 unmodified lines | ||
| 5344 | 5347 | if (queue_dp) |
| 5345 | 5348 | schedule_work(&rdev->dp_work); |
| 5346 | 5349 | if (queue_hotplug) |
| 5347 | − schedule_work(&rdev->hotplug_work); | |
| 5350 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 5348 | 5351 | if (queue_hdmi) |
| 5349 | 5352 | schedule_work(&rdev->audio_work); |
| 5350 | 5353 | if (queue_thermal && rdev->pm.dpm_enabled) |
| @@ -806,7 +806,7 @@ int r100_irq_process(struct radeon_device *rdev) · 805 unmodified lines | ||
| 806 | 806 | status = r100_irq_ack(rdev); |
| 807 | 807 | } |
| 808 | 808 | if (queue_hotplug) |
| 809 | − schedule_work(&rdev->hotplug_work); | |
| 809 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 810 | 810 | if (rdev->msi_enabled) { |
| 811 | 811 | switch (rdev->family) { |
| 812 | 812 | case CHIP_RS400: |
| @@ -3217,6 +3217,9 @@ void r100_bandwidth_update(struct radeon_device *rdev) · 2404 unmodified lines | ||
| 3217 | 3217 | uint32_t pixel_bytes1 = 0; |
| 3218 | 3218 | uint32_t pixel_bytes2 = 0; |
| 3219 | 3219 | |
| 3220 | + /* Guess line buffer size to be 8192 pixels */ | |
| 3221 | + u32 lb_size = 8192; | |
| 3222 | + | |
| 3220 | 3223 | if (!rdev->mode_info.mode_config_initialized) |
| 3221 | 3224 | return; |
| 3222 | 3225 | |
| @@ -3631,6 +3634,13 @@ void r100_bandwidth_update(struct radeon_device *rdev) · 408 unmodified lines | ||
| 3631 | 3634 | DRM_DEBUG_KMS("GRPH2_BUFFER_CNTL from to %x\n", |
| 3632 | 3635 | (unsigned int)RREG32(RADEON_GRPH2_BUFFER_CNTL)); |
| 3633 | 3636 | } |
| 3637 | + | |
| 3638 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 3639 | + if (mode1) | |
| 3640 | + rdev->mode_info.crtcs[0]->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode1->crtc_hdisplay); | |
| 3641 | + | |
| 3642 | + if (mode2) | |
| 3643 | + rdev->mode_info.crtcs[1]->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode2->crtc_hdisplay); | |
| 3634 | 3644 | } |
| 3635 | 3645 | |
| 3636 | 3646 | int r100_ring_test(struct radeon_device *rdev, struct radeon_ring *ring) |
| @@ -4276,7 +4276,7 @@ restart_ih: · 4275 unmodified lines | ||
| 4276 | 4276 | WREG32(IH_RB_RPTR, rptr); |
| 4277 | 4277 | } |
| 4278 | 4278 | if (queue_hotplug) |
| 4279 | − schedule_work(&rdev->hotplug_work); | |
| 4279 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 4280 | 4280 | if (queue_hdmi) |
| 4281 | 4281 | schedule_work(&rdev->audio_work); |
| 4282 | 4282 | if (queue_thermal && rdev->pm.dpm_enabled) |
| @@ -2414,7 +2414,7 @@ struct radeon_device { · 2413 unmodified lines | ||
| 2414 | 2414 | struct r600_ih ih; /* r6/700 interrupt ring */ |
| 2415 | 2415 | struct radeon_rlc rlc; |
| 2416 | 2416 | struct radeon_mec mec; |
| 2417 | − struct work_struct hotplug_work; | |
| 2417 | + struct delayed_work hotplug_work; | |
| 2418 | 2418 | struct work_struct dp_work; |
| 2419 | 2419 | struct work_struct audio_work; |
| 2420 | 2420 | int num_crtc; /* number of crtcs */ |
| @@ -54,6 +54,9 @@ static struct radeon_agpmode_quirk radeon_agpmode_quirk_list[] = { · 53 unmodified lines | ||
| 54 | 54 | /* Intel 82855PM host bridge / Mobility 9600 M10 RV350 Needs AGPMode 1 (lp #195051) */ |
| 55 | 55 | { PCI_VENDOR_ID_INTEL, 0x3340, PCI_VENDOR_ID_ATI, 0x4e50, |
| 56 | 56 | PCI_VENDOR_ID_IBM, 0x0550, 1}, |
| 57 | + /* Intel 82855PM host bridge / RV250/M9 GL [Mobility FireGL 9000/Radeon 9000] needs AGPMode 1 (Thinkpad T40p) */ | |
| 58 | + { PCI_VENDOR_ID_INTEL, 0x3340, PCI_VENDOR_ID_ATI, 0x4c66, | |
| 59 | + PCI_VENDOR_ID_IBM, 0x054d, 1}, | |
| 57 | 60 | /* Intel 82855PM host bridge / Mobility M7 needs AGPMode 1 */ |
| 58 | 61 | { PCI_VENDOR_ID_INTEL, 0x3340, PCI_VENDOR_ID_ATI, 0x4c57, |
| 59 | 62 | PCI_VENDOR_ID_IBM, 0x0530, 1}, |
| @@ -1234,13 +1234,32 @@ radeon_dvi_detect(struct drm_connector *connector, bool force) · 1233 unmodified lines | ||
| 1234 | 1234 | if (r < 0) |
| 1235 | 1235 | return connector_status_disconnected; |
| 1236 | 1236 | |
| 1237 | + if (radeon_connector->detected_hpd_without_ddc) { | |
| 1238 | + force = true; | |
| 1239 | + radeon_connector->detected_hpd_without_ddc = false; | |
| 1240 | + } | |
| 1241 | + | |
| 1237 | 1242 | if (!force && radeon_check_hpd_status_unchanged(connector)) { |
| 1238 | 1243 | ret = connector->status; |
| 1239 | 1244 | goto exit; |
| 1240 | 1245 | } |
| 1241 | 1246 | |
| 1242 | − if (radeon_connector->ddc_bus) | |
| 1247 | + if (radeon_connector->ddc_bus) { | |
| 1243 | 1248 | dret = radeon_ddc_probe(radeon_connector, false); |
| 1249 | + | |
| 1250 | + /* Sometimes the pins required for the DDC probe on DVI | |
| 1251 | + * connectors don't make contact at the same time that the ones | |
| 1252 | + * for HPD do. If the DDC probe fails even though we had an HPD | |
| 1253 | + * signal, try again later */ | |
| 1254 | + if (!dret && !force && | |
| 1255 | + connector->status != connector_status_connected) { | |
| 1256 | + DRM_DEBUG_KMS("hpd detected without ddc, retrying in 1 second\n"); | |
| 1257 | + radeon_connector->detected_hpd_without_ddc = true; | |
| 1258 | + schedule_delayed_work(&rdev->hotplug_work, | |
| 1259 | + msecs_to_jiffies(1000)); | |
| 1260 | + goto exit; | |
| 1261 | + } | |
| 1262 | + } | |
| 1244 | 1263 | if (dret) { |
| 1245 | 1264 | radeon_connector->detected_by_load = false; |
| 1246 | 1265 | radeon_connector_free_edid(connector); |
| @@ -322,7 +322,9 @@ void radeon_crtc_handle_vblank(struct radeon_device *rdev, int crtc_id) · 321 unmodified lines | ||
| 322 | 322 | * to complete in this vblank? |
| 323 | 323 | */ |
| 324 | 324 | if (update_pending && |
| 325 | − (DRM_SCANOUTPOS_VALID & radeon_get_crtc_scanoutpos(rdev->ddev, crtc_id, 0, | |
| 325 | + (DRM_SCANOUTPOS_VALID & radeon_get_crtc_scanoutpos(rdev->ddev, | |
| 326 | + crtc_id, | |
| 327 | + USE_REAL_VBLANKSTART, | |
| 326 | 328 | &vpos, &hpos, NULL, NULL, |
| 327 | 329 | &rdev->mode_info.crtcs[crtc_id]->base.hwmode)) && |
| 328 | 330 | ((vpos >= (99 * rdev->mode_info.crtcs[crtc_id]->base.hwmode.crtc_vdisplay)/100) || |
| @@ -401,6 +403,8 @@ static void radeon_flip_work_func(struct work_struct *__work) · 72 unmodified lines | ||
| 401 | 403 | struct drm_crtc *crtc = &radeon_crtc->base; |
| 402 | 404 | unsigned long flags; |
| 403 | 405 | int r; |
| 406 | + int vpos, hpos, stat, min_udelay; | |
| 407 | + struct drm_vblank_crtc *vblank = &crtc->dev->vblank[work->crtc_id]; | |
| 404 | 408 | |
| 405 | 409 | down_read(&rdev->exclusive_lock); |
| 406 | 410 | if (work->fence) { |
| @@ -437,6 +441,41 @@ static void radeon_flip_work_func(struct work_struct *__work) · 30 unmodified lines | ||
| 437 | 441 | /* set the proper interrupt */ |
| 438 | 442 | radeon_irq_kms_pflip_irq_get(rdev, radeon_crtc->crtc_id); |
| 439 | 443 | |
| 444 | + /* If this happens to execute within the "virtually extended" vblank | |
| 445 | + * interval before the start of the real vblank interval then it needs | |
| 446 | + * to delay programming the mmio flip until the real vblank is entered. | |
| 447 | + * This prevents completing a flip too early due to the way we fudge | |
| 448 | + * our vblank counter and vblank timestamps in order to work around the | |
| 449 | + * problem that the hw fires vblank interrupts before actual start of | |
| 450 | + * vblank (when line buffer refilling is done for a frame). It | |
| 451 | + * complements the fudging logic in radeon_get_crtc_scanoutpos() for | |
| 452 | + * timestamping and radeon_get_vblank_counter_kms() for vblank counts. | |
| 453 | + * | |
| 454 | + * In practice this won't execute very often unless on very fast | |
| 455 | + * machines because the time window for this to happen is very small. | |
| 456 | + */ | |
| 457 | + for (;;) { | |
| 458 | + /* GET_DISTANCE_TO_VBLANKSTART returns distance to real vblank | |
| 459 | + * start in hpos, and to the "fudged earlier" vblank start in | |
| 460 | + * vpos. | |
| 461 | + */ | |
| 462 | + stat = radeon_get_crtc_scanoutpos(rdev->ddev, work->crtc_id, | |
| 463 | + GET_DISTANCE_TO_VBLANKSTART, | |
| 464 | + &vpos, &hpos, NULL, NULL, | |
| 465 | + &crtc->hwmode); | |
| 466 | + | |
| 467 | + if ((stat & (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE)) != | |
| 468 | + (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE) || | |
| 469 | + !(vpos >= 0 && hpos <= 0)) | |
| 470 | + break; | |
| 471 | + | |
| 472 | + /* Sleep at least until estimated real start of hw vblank */ | |
| 473 | + spin_unlock_irqrestore(&crtc->dev->event_lock, flags); | |
| 474 | + min_udelay = (-hpos + 1) * max(vblank->linedur_ns / 1000, 5); | |
| 475 | + usleep_range(min_udelay, 2 * min_udelay); | |
| 476 | + spin_lock_irqsave(&crtc->dev->event_lock, flags); | |
| 477 | + }; | |
| 478 | + | |
| 440 | 479 | /* do the flip (mmio) */ |
| 441 | 480 | radeon_page_flip(rdev, radeon_crtc->crtc_id, work->base); |
| 442 | 481 | |
| @@ -1768,6 +1807,15 @@ bool radeon_crtc_scaling_mode_fixup(struct drm_crtc *crtc, · 1325 unmodified lines | ||
| 1768 | 1807 | * \param dev Device to query. |
| 1769 | 1808 | * \param crtc Crtc to query. |
| 1770 | 1809 | * \param flags Flags from caller (DRM_CALLED_FROM_VBLIRQ or 0). |
| 1810 | + * For driver internal use only also supports these flags: | |
| 1811 | + * | |
| 1812 | + * USE_REAL_VBLANKSTART to use the real start of vblank instead | |
| 1813 | + * of a fudged earlier start of vblank. | |
| 1814 | + * | |
| 1815 | + * GET_DISTANCE_TO_VBLANKSTART to return distance to the | |
| 1816 | + * fudged earlier start of vblank in *vpos and the distance | |
| 1817 | + * to true start of vblank in *hpos. | |
| 1818 | + * | |
| 1771 | 1819 | * \param *vpos Location where vertical scanout position should be stored. |
| 1772 | 1820 | * \param *hpos Location where horizontal scanout position should go. |
| 1773 | 1821 | * \param *stime Target location for timestamp taken immediately before |
| @@ -1911,10 +1959,40 @@ int radeon_get_crtc_scanoutpos(struct drm_device *dev, unsigned int pipe, · 137 unmodified lines | ||
| 1911 | 1959 | vbl_end = 0; |
| 1912 | 1960 | } |
| 1913 | 1961 | |
| 1962 | + /* Called from driver internal vblank counter query code? */ | |
| 1963 | + if (flags & GET_DISTANCE_TO_VBLANKSTART) { | |
| 1964 | + /* Caller wants distance from real vbl_start in *hpos */ | |
| 1965 | + *hpos = *vpos - vbl_start; | |
| 1966 | + } | |
| 1967 | + | |
| 1968 | + /* Fudge vblank to start a few scanlines earlier to handle the | |
| 1969 | + * problem that vblank irqs fire a few scanlines before start | |
| 1970 | + * of vblank. Some driver internal callers need the true vblank | |
| 1971 | + * start to be used and signal this via the USE_REAL_VBLANKSTART flag. | |
| 1972 | + * | |
| 1973 | + * The cause of the "early" vblank irq is that the irq is triggered | |
| 1974 | + * by the line buffer logic when the line buffer read position enters | |
| 1975 | + * the vblank, whereas our crtc scanout position naturally lags the | |
| 1976 | + * line buffer read position. | |
| 1977 | + */ | |
| 1978 | + if (!(flags & USE_REAL_VBLANKSTART)) | |
| 1979 | + vbl_start -= rdev->mode_info.crtcs[pipe]->lb_vblank_lead_lines; | |
| 1980 | + | |
| 1914 | 1981 | /* Test scanout position against vblank region. */ |
| 1915 | 1982 | if ((*vpos < vbl_start) && (*vpos >= vbl_end)) |
| 1916 | 1983 | in_vbl = false; |
| 1917 | 1984 | |
| 1985 | + /* In vblank? */ | |
| 1986 | + if (in_vbl) | |
| 1987 | + ret |= DRM_SCANOUTPOS_IN_VBLANK; | |
| 1988 | + | |
| 1989 | + /* Called from driver internal vblank counter query code? */ | |
| 1990 | + if (flags & GET_DISTANCE_TO_VBLANKSTART) { | |
| 1991 | + /* Caller wants distance from fudged earlier vbl_start */ | |
| 1992 | + *vpos -= vbl_start; | |
| 1993 | + return ret; | |
| 1994 | + } | |
| 1995 | + | |
| 1918 | 1996 | /* Check if inside vblank area and apply corrective offsets: |
| 1919 | 1997 | * vpos will then be >=0 in video scanout area, but negative |
| 1920 | 1998 | * within vblank area, counting down the number of lines until |
| @@ -1930,31 +2008,5 @@ int radeon_get_crtc_scanoutpos(struct drm_device *dev, unsigned int pipe, · 9 unmodified lines | ||
| 1930 | 2008 | /* Correct for shifted end of vbl at vbl_end. */ |
| 1931 | 2009 | *vpos = *vpos - vbl_end; |
| 1932 | 2010 | |
| 1933 | − /* In vblank? */ | |
| 1934 | − if (in_vbl) | |
| 1935 | − ret |= DRM_SCANOUTPOS_IN_VBLANK; | |
| 1936 | − | |
| 1937 | − /* Is vpos outside nominal vblank area, but less than | |
| 1938 | − * 1/100 of a frame height away from start of vblank? | |
| 1939 | − * If so, assume this isn't a massively delayed vblank | |
| 1940 | − * interrupt, but a vblank interrupt that fired a few | |
| 1941 | − * microseconds before true start of vblank. Compensate | |
| 1942 | − * by adding a full frame duration to the final timestamp. | |
| 1943 | − * Happens, e.g., on ATI R500, R600. | |
| 1944 | − * | |
| 1945 | − * We only do this if DRM_CALLED_FROM_VBLIRQ. | |
| 1946 | − */ | |
| 1947 | − if ((flags & DRM_CALLED_FROM_VBLIRQ) && !in_vbl) { | |
| 1948 | − vbl_start = mode->crtc_vdisplay; | |
| 1949 | − vtotal = mode->crtc_vtotal; | |
| 1950 | − | |
| 1951 | − if (vbl_start - *vpos < vtotal / 100) { | |
| 1952 | − *vpos -= vtotal; | |
| 1953 | − | |
| 1954 | − /* Signal this correction as "applied". */ | |
| 1955 | − ret |= 0x8; | |
| 1956 | − } | |
| 1957 | − } | |
| 1958 | − | |
| 1959 | 2011 | return ret; |
| 1960 | 2012 | } |
| @@ -74,7 +74,7 @@ irqreturn_t radeon_driver_irq_handler_kms(int irq, void *arg) · 73 unmodified lines | ||
| 74 | 74 | static void radeon_hotplug_work_func(struct work_struct *work) |
| 75 | 75 | { |
| 76 | 76 | struct radeon_device *rdev = container_of(work, struct radeon_device, |
| 77 | − hotplug_work); | |
| 77 | + hotplug_work.work); | |
| 78 | 78 | struct drm_device *dev = rdev->ddev; |
| 79 | 79 | struct drm_mode_config *mode_config = &dev->mode_config; |
| 80 | 80 | struct drm_connector *connector; |
| @@ -302,7 +302,7 @@ int radeon_irq_kms_init(struct radeon_device *rdev) · 221 unmodified lines | ||
| 302 | 302 | } |
| 303 | 303 | } |
| 304 | 304 | |
| 305 | − INIT_WORK(&rdev->hotplug_work, radeon_hotplug_work_func); | |
| 305 | + INIT_DELAYED_WORK(&rdev->hotplug_work, radeon_hotplug_work_func); | |
| 306 | 306 | INIT_WORK(&rdev->dp_work, radeon_dp_work_func); |
| 307 | 307 | INIT_WORK(&rdev->audio_work, r600_audio_update_hdmi); |
| 308 | 308 | |
| @@ -310,7 +310,7 @@ int radeon_irq_kms_init(struct radeon_device *rdev) · 1 unmodified line | ||
| 310 | 310 | r = drm_irq_install(rdev->ddev, rdev->ddev->pdev->irq); |
| 311 | 311 | if (r) { |
| 312 | 312 | rdev->irq.installed = false; |
| 313 | − flush_work(&rdev->hotplug_work); | |
| 313 | + flush_delayed_work(&rdev->hotplug_work); | |
| 314 | 314 | return r; |
| 315 | 315 | } |
| 316 | 316 | |
| @@ -333,7 +333,7 @@ void radeon_irq_kms_fini(struct radeon_device *rdev) · 16 unmodified lines | ||
| 333 | 333 | rdev->irq.installed = false; |
| 334 | 334 | if (rdev->msi_enabled) |
| 335 | 335 | pci_disable_msi(rdev->pdev); |
| 336 | − flush_work(&rdev->hotplug_work); | |
| 336 | + flush_delayed_work(&rdev->hotplug_work); | |
| 337 | 337 | } |
| 338 | 338 | } |
| 339 | 339 | |
| @@ -755,6 +755,8 @@ void radeon_driver_preclose_kms(struct drm_device *dev, · 754 unmodified lines | ||
| 755 | 755 | */ |
| 756 | 756 | u32 radeon_get_vblank_counter_kms(struct drm_device *dev, int crtc) |
| 757 | 757 | { |
| 758 | + int vpos, hpos, stat; | |
| 759 | + u32 count; | |
| 758 | 760 | struct radeon_device *rdev = dev->dev_private; |
| 759 | 761 | |
| 760 | 762 | if (crtc < 0 || crtc >= rdev->num_crtc) { |
| @@ -762,7 +764,53 @@ u32 radeon_get_vblank_counter_kms(struct drm_device *dev, int crtc) · 1 unmodified line | ||
| 762 | 764 | return -EINVAL; |
| 763 | 765 | } |
| 764 | 766 | |
| 765 | − return radeon_get_vblank_counter(rdev, crtc); | |
| 767 | + /* The hw increments its frame counter at start of vsync, not at start | |
| 768 | + * of vblank, as is required by DRM core vblank counter handling. | |
| 769 | + * Cook the hw count here to make it appear to the caller as if it | |
| 770 | + * incremented at start of vblank. We measure distance to start of | |
| 771 | + * vblank in vpos. vpos therefore will be >= 0 between start of vblank | |
| 772 | + * and start of vsync, so vpos >= 0 means to bump the hw frame counter | |
| 773 | + * result by 1 to give the proper appearance to caller. | |
| 774 | + */ | |
| 775 | + if (rdev->mode_info.crtcs[crtc]) { | |
| 776 | + /* Repeat readout if needed to provide stable result if | |
| 777 | + * we cross start of vsync during the queries. | |
| 778 | + */ | |
| 779 | + do { | |
| 780 | + count = radeon_get_vblank_counter(rdev, crtc); | |
| 781 | + /* Ask radeon_get_crtc_scanoutpos to return vpos as | |
| 782 | + * distance to start of vblank, instead of regular | |
| 783 | + * vertical scanout pos. | |
| 784 | + */ | |
| 785 | + stat = radeon_get_crtc_scanoutpos( | |
| 786 | + dev, crtc, GET_DISTANCE_TO_VBLANKSTART, | |
| 787 | + &vpos, &hpos, NULL, NULL, | |
| 788 | + &rdev->mode_info.crtcs[crtc]->base.hwmode); | |
| 789 | + } while (count != radeon_get_vblank_counter(rdev, crtc)); | |
| 790 | + | |
| 791 | + if (((stat & (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE)) != | |
| 792 | + (DRM_SCANOUTPOS_VALID | DRM_SCANOUTPOS_ACCURATE))) { | |
| 793 | + DRM_DEBUG_VBL("Query failed! stat %d\n", stat); | |
| 794 | + } | |
| 795 | + else { | |
| 796 | + DRM_DEBUG_VBL("crtc %d: dist from vblank start %d\n", | |
| 797 | + crtc, vpos); | |
| 798 | + | |
| 799 | + /* Bump counter if we are at >= leading edge of vblank, | |
| 800 | + * but before vsync where vpos would turn negative and | |
| 801 | + * the hw counter really increments. | |
| 802 | + */ | |
| 803 | + if (vpos >= 0) | |
| 804 | + count++; | |
| 805 | + } | |
| 806 | + } | |
| 807 | + else { | |
| 808 | + /* Fallback to use value as is. */ | |
| 809 | + count = radeon_get_vblank_counter(rdev, crtc); | |
| 810 | + DRM_DEBUG_VBL("NULL mode info! Returned count may be wrong.\n"); | |
| 811 | + } | |
| 812 | + | |
| 813 | + return count; | |
| 766 | 814 | } |
| 767 | 815 | |
| 768 | 816 | /** |
| @@ -367,6 +367,7 @@ struct radeon_crtc { · 366 unmodified lines | ||
| 367 | 367 | u32 line_time; |
| 368 | 368 | u32 wm_low; |
| 369 | 369 | u32 wm_high; |
| 370 | + u32 lb_vblank_lead_lines; | |
| 370 | 371 | struct drm_display_mode hw_mode; |
| 371 | 372 | enum radeon_output_csc output_csc; |
| 372 | 373 | }; |
| @@ -553,6 +554,7 @@ struct radeon_connector { · 180 unmodified lines | ||
| 553 | 554 | void *con_priv; |
| 554 | 555 | bool dac_load_detect; |
| 555 | 556 | bool detected_by_load; /* if the connection status was determined by load */ |
| 557 | + bool detected_hpd_without_ddc; /* if an HPD signal was detected on DVI, but ddc probing failed */ | |
| 556 | 558 | uint16_t connector_object_id; |
| 557 | 559 | struct radeon_hpd hpd; |
| 558 | 560 | struct radeon_router router; |
| @@ -686,6 +688,9 @@ struct atom_voltage_table · 127 unmodified lines | ||
| 686 | 688 | struct atom_voltage_table_entry entries[MAX_VOLTAGE_ENTRIES]; |
| 687 | 689 | }; |
| 688 | 690 | |
| 691 | +/* Driver internal use only flags of radeon_get_crtc_scanoutpos() */ | |
| 692 | +#define USE_REAL_VBLANKSTART (1 << 30) | |
| 693 | +#define GET_DISTANCE_TO_VBLANKSTART (1 << 31) | |
| 689 | 694 | |
| 690 | 695 | extern void |
| 691 | 696 | radeon_add_atom_connector(struct drm_device *dev, |
| @@ -1756,7 +1756,9 @@ static bool radeon_pm_in_vbl(struct radeon_device *rdev) · 1755 unmodified lines | ||
| 1756 | 1756 | */ |
| 1757 | 1757 | for (crtc = 0; (crtc < rdev->num_crtc) && in_vbl; crtc++) { |
| 1758 | 1758 | if (rdev->pm.active_crtcs & (1 << crtc)) { |
| 1759 | − vbl_status = radeon_get_crtc_scanoutpos(rdev->ddev, crtc, 0, | |
| 1759 | + vbl_status = radeon_get_crtc_scanoutpos(rdev->ddev, | |
| 1760 | + crtc, | |
| 1761 | + USE_REAL_VBLANKSTART, | |
| 1760 | 1762 | &vpos, &hpos, NULL, NULL, |
| 1761 | 1763 | &rdev->mode_info.crtcs[crtc]->base.hwmode); |
| 1762 | 1764 | if ((vbl_status & DRM_SCANOUTPOS_VALID) && |
| @@ -361,31 +361,31 @@ int radeon_vce_get_create_msg(struct radeon_device *rdev, int ring, · 360 unmodified lines | ||
| 361 | 361 | |
| 362 | 362 | /* stitch together an VCE create msg */ |
| 363 | 363 | ib.length_dw = 0; |
| 364 | − ib.ptr[ib.length_dw++] = 0x0000000c; /* len */ | |
| 365 | − ib.ptr[ib.length_dw++] = 0x00000001; /* session cmd */ | |
| 366 | − ib.ptr[ib.length_dw++] = handle; | |
| 367 | − | |
| 368 | − ib.ptr[ib.length_dw++] = 0x00000030; /* len */ | |
| 369 | − ib.ptr[ib.length_dw++] = 0x01000001; /* create cmd */ | |
| 370 | − ib.ptr[ib.length_dw++] = 0x00000000; | |
| 371 | − ib.ptr[ib.length_dw++] = 0x00000042; | |
| 372 | − ib.ptr[ib.length_dw++] = 0x0000000a; | |
| 373 | − ib.ptr[ib.length_dw++] = 0x00000001; | |
| 374 | − ib.ptr[ib.length_dw++] = 0x00000080; | |
| 375 | − ib.ptr[ib.length_dw++] = 0x00000060; | |
| 376 | − ib.ptr[ib.length_dw++] = 0x00000100; | |
| 377 | − ib.ptr[ib.length_dw++] = 0x00000100; | |
| 378 | − ib.ptr[ib.length_dw++] = 0x0000000c; | |
| 379 | − ib.ptr[ib.length_dw++] = 0x00000000; | |
| 380 | − | |
| 381 | − ib.ptr[ib.length_dw++] = 0x00000014; /* len */ | |
| 382 | − ib.ptr[ib.length_dw++] = 0x05000005; /* feedback buffer */ | |
| 383 | − ib.ptr[ib.length_dw++] = upper_32_bits(dummy); | |
| 384 | − ib.ptr[ib.length_dw++] = dummy; | |
| 385 | − ib.ptr[ib.length_dw++] = 0x00000001; | |
| 364 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x0000000c); /* len */ | |
| 365 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000001); /* session cmd */ | |
| 366 | + ib.ptr[ib.length_dw++] = cpu_to_le32(handle); | |
| 367 | + | |
| 368 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000030); /* len */ | |
| 369 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x01000001); /* create cmd */ | |
| 370 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000000); | |
| 371 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000042); | |
| 372 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x0000000a); | |
| 373 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000001); | |
| 374 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000080); | |
| 375 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000060); | |
| 376 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000100); | |
| 377 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000100); | |
| 378 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x0000000c); | |
| 379 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000000); | |
| 380 | + | |
| 381 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000014); /* len */ | |
| 382 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x05000005); /* feedback buffer */ | |
| 383 | + ib.ptr[ib.length_dw++] = cpu_to_le32(upper_32_bits(dummy)); | |
| 384 | + ib.ptr[ib.length_dw++] = cpu_to_le32(dummy); | |
| 385 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000001); | |
| 386 | 386 | |
| 387 | 387 | for (i = ib.length_dw; i < ib_size_dw; ++i) |
| 388 | − ib.ptr[i] = 0x0; | |
| 388 | + ib.ptr[i] = cpu_to_le32(0x0); | |
| 389 | 389 | |
| 390 | 390 | r = radeon_ib_schedule(rdev, &ib, NULL, false); |
| 391 | 391 | if (r) { |
| @@ -428,21 +428,21 @@ int radeon_vce_get_destroy_msg(struct radeon_device *rdev, int ring, · 36 unmodified lines | ||
| 428 | 428 | |
| 429 | 429 | /* stitch together an VCE destroy msg */ |
| 430 | 430 | ib.length_dw = 0; |
| 431 | − ib.ptr[ib.length_dw++] = 0x0000000c; /* len */ | |
| 432 | − ib.ptr[ib.length_dw++] = 0x00000001; /* session cmd */ | |
| 433 | − ib.ptr[ib.length_dw++] = handle; | |
| 431 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x0000000c); /* len */ | |
| 432 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000001); /* session cmd */ | |
| 433 | + ib.ptr[ib.length_dw++] = cpu_to_le32(handle); | |
| 434 | 434 | |
| 435 | − ib.ptr[ib.length_dw++] = 0x00000014; /* len */ | |
| 436 | − ib.ptr[ib.length_dw++] = 0x05000005; /* feedback buffer */ | |
| 437 | − ib.ptr[ib.length_dw++] = upper_32_bits(dummy); | |
| 438 | − ib.ptr[ib.length_dw++] = dummy; | |
| 439 | − ib.ptr[ib.length_dw++] = 0x00000001; | |
| 435 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000014); /* len */ | |
| 436 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x05000005); /* feedback buffer */ | |
| 437 | + ib.ptr[ib.length_dw++] = cpu_to_le32(upper_32_bits(dummy)); | |
| 438 | + ib.ptr[ib.length_dw++] = cpu_to_le32(dummy); | |
| 439 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000001); | |
| 440 | 440 | |
| 441 | − ib.ptr[ib.length_dw++] = 0x00000008; /* len */ | |
| 442 | − ib.ptr[ib.length_dw++] = 0x02000001; /* destroy cmd */ | |
| 441 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x00000008); /* len */ | |
| 442 | + ib.ptr[ib.length_dw++] = cpu_to_le32(0x02000001); /* destroy cmd */ | |
| 443 | 443 | |
| 444 | 444 | for (i = ib.length_dw; i < ib_size_dw; ++i) |
| 445 | − ib.ptr[i] = 0x0; | |
| 445 | + ib.ptr[i] = cpu_to_le32(0x0); | |
| 446 | 446 | |
| 447 | 447 | r = radeon_ib_schedule(rdev, &ib, NULL, false); |
| 448 | 448 | if (r) { |
| @@ -699,12 +699,12 @@ bool radeon_vce_semaphore_emit(struct radeon_device *rdev, · 250 unmodified lines | ||
| 699 | 699 | { |
| 700 | 700 | uint64_t addr = semaphore->gpu_addr; |
| 701 | 701 | |
| 702 | − radeon_ring_write(ring, VCE_CMD_SEMAPHORE); | |
| 703 | − radeon_ring_write(ring, (addr >> 3) & 0x000FFFFF); | |
| 704 | − radeon_ring_write(ring, (addr >> 23) & 0x000FFFFF); | |
| 705 | − radeon_ring_write(ring, 0x01003000 | (emit_wait ? 1 : 0)); | |
| 702 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_SEMAPHORE)); | |
| 703 | + radeon_ring_write(ring, cpu_to_le32((addr >> 3) & 0x000FFFFF)); | |
| 704 | + radeon_ring_write(ring, cpu_to_le32((addr >> 23) & 0x000FFFFF)); | |
| 705 | + radeon_ring_write(ring, cpu_to_le32(0x01003000 | (emit_wait ? 1 : 0))); | |
| 706 | 706 | if (!emit_wait) |
| 707 | − radeon_ring_write(ring, VCE_CMD_END); | |
| 707 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_END)); | |
| 708 | 708 | |
| 709 | 709 | return true; |
| 710 | 710 | } |
| @@ -719,10 +719,10 @@ bool radeon_vce_semaphore_emit(struct radeon_device *rdev, · 8 unmodified lines | ||
| 719 | 719 | void radeon_vce_ib_execute(struct radeon_device *rdev, struct radeon_ib *ib) |
| 720 | 720 | { |
| 721 | 721 | struct radeon_ring *ring = &rdev->ring[ib->ring]; |
| 722 | − radeon_ring_write(ring, VCE_CMD_IB); | |
| 723 | − radeon_ring_write(ring, ib->gpu_addr); | |
| 724 | − radeon_ring_write(ring, upper_32_bits(ib->gpu_addr)); | |
| 725 | − radeon_ring_write(ring, ib->length_dw); | |
| 722 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_IB)); | |
| 723 | + radeon_ring_write(ring, cpu_to_le32(ib->gpu_addr)); | |
| 724 | + radeon_ring_write(ring, cpu_to_le32(upper_32_bits(ib->gpu_addr))); | |
| 725 | + radeon_ring_write(ring, cpu_to_le32(ib->length_dw)); | |
| 726 | 726 | } |
| 727 | 727 | |
| 728 | 728 | /** |
| @@ -738,12 +738,12 @@ void radeon_vce_fence_emit(struct radeon_device *rdev, · 9 unmodified lines | ||
| 738 | 738 | struct radeon_ring *ring = &rdev->ring[fence->ring]; |
| 739 | 739 | uint64_t addr = rdev->fence_drv[fence->ring].gpu_addr; |
| 740 | 740 | |
| 741 | − radeon_ring_write(ring, VCE_CMD_FENCE); | |
| 742 | − radeon_ring_write(ring, addr); | |
| 743 | − radeon_ring_write(ring, upper_32_bits(addr)); | |
| 744 | − radeon_ring_write(ring, fence->seq); | |
| 745 | − radeon_ring_write(ring, VCE_CMD_TRAP); | |
| 746 | − radeon_ring_write(ring, VCE_CMD_END); | |
| 741 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_FENCE)); | |
| 742 | + radeon_ring_write(ring, cpu_to_le32(addr)); | |
| 743 | + radeon_ring_write(ring, cpu_to_le32(upper_32_bits(addr))); | |
| 744 | + radeon_ring_write(ring, cpu_to_le32(fence->seq)); | |
| 745 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_TRAP)); | |
| 746 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_END)); | |
| 747 | 747 | } |
| 748 | 748 | |
| 749 | 749 | /** |
| @@ -765,7 +765,7 @@ int radeon_vce_ring_test(struct radeon_device *rdev, struct radeon_ring *ring) · 15 unmodified lines | ||
| 765 | 765 | ring->idx, r); |
| 766 | 766 | return r; |
| 767 | 767 | } |
| 768 | − radeon_ring_write(ring, VCE_CMD_END); | |
| 768 | + radeon_ring_write(ring, cpu_to_le32(VCE_CMD_END)); | |
| 769 | 769 | radeon_ring_unlock_commit(rdev, ring, false); |
| 770 | 770 | |
| 771 | 771 | for (i = 0; i < rdev->usec_timeout; i++) { |
| @@ -813,7 +813,7 @@ int rs600_irq_process(struct radeon_device *rdev) · 812 unmodified lines | ||
| 813 | 813 | status = rs600_irq_ack(rdev); |
| 814 | 814 | } |
| 815 | 815 | if (queue_hotplug) |
| 816 | − schedule_work(&rdev->hotplug_work); | |
| 816 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 817 | 817 | if (queue_hdmi) |
| 818 | 818 | schedule_work(&rdev->audio_work); |
| 819 | 819 | if (rdev->msi_enabled) { |
| @@ -207,6 +207,9 @@ void rs690_line_buffer_adjust(struct radeon_device *rdev, · 206 unmodified lines | ||
| 207 | 207 | { |
| 208 | 208 | u32 tmp; |
| 209 | 209 | |
| 210 | + /* Guess line buffer size to be 8192 pixels */ | |
| 211 | + u32 lb_size = 8192; | |
| 212 | + | |
| 210 | 213 | /* |
| 211 | 214 | * Line Buffer Setup |
| 212 | 215 | * There is a single line buffer shared by both display controllers. |
| @@ -243,6 +246,13 @@ void rs690_line_buffer_adjust(struct radeon_device *rdev, · 30 unmodified lines | ||
| 243 | 246 | tmp |= V_006520_DC_LB_MEMORY_SPLIT_D1_1Q_D2_3Q; |
| 244 | 247 | } |
| 245 | 248 | WREG32(R_006520_DC_LB_MEMORY_SPLIT, tmp); |
| 249 | + | |
| 250 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 251 | + if (mode1) | |
| 252 | + rdev->mode_info.crtcs[0]->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode1->crtc_hdisplay); | |
| 253 | + | |
| 254 | + if (mode2) | |
| 255 | + rdev->mode_info.crtcs[1]->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode2->crtc_hdisplay); | |
| 246 | 256 | } |
| 247 | 257 | |
| 248 | 258 | struct rs690_watermark { |
| @@ -464,7 +464,7 @@ void rv730_stop_dpm(struct radeon_device *rdev) · 463 unmodified lines | ||
| 464 | 464 | result = rv770_send_msg_to_smc(rdev, PPSMC_MSG_TwoLevelsDisabled); |
| 465 | 465 | |
| 466 | 466 | if (result != PPSMC_Result_OK) |
| 467 | − DRM_ERROR("Could not force DPM to low\n"); | |
| 467 | + DRM_DEBUG("Could not force DPM to low\n"); | |
| 468 | 468 | |
| 469 | 469 | WREG32_P(GENERAL_PWRMGT, 0, ~GLOBAL_PWRMGT_EN); |
| 470 | 470 | |
| @@ -193,7 +193,7 @@ void rv770_stop_dpm(struct radeon_device *rdev) · 192 unmodified lines | ||
| 193 | 193 | result = rv770_send_msg_to_smc(rdev, PPSMC_MSG_TwoLevelsDisabled); |
| 194 | 194 | |
| 195 | 195 | if (result != PPSMC_Result_OK) |
| 196 | − DRM_ERROR("Could not force DPM to low.\n"); | |
| 196 | + DRM_DEBUG("Could not force DPM to low.\n"); | |
| 197 | 197 | |
| 198 | 198 | WREG32_P(GENERAL_PWRMGT, 0, ~GLOBAL_PWRMGT_EN); |
| 199 | 199 | |
| @@ -1418,7 +1418,7 @@ int rv770_resume_smc(struct radeon_device *rdev) · 1218 unmodified lines | ||
| 1418 | 1418 | int rv770_set_sw_state(struct radeon_device *rdev) |
| 1419 | 1419 | { |
| 1420 | 1420 | if (rv770_send_msg_to_smc(rdev, PPSMC_MSG_SwitchToSwState) != PPSMC_Result_OK) |
| 1421 | − return -EINVAL; | |
| 1421 | + DRM_DEBUG("rv770_set_sw_state failed\n"); | |
| 1422 | 1422 | return 0; |
| 1423 | 1423 | } |
| 1424 | 1424 | |
| @@ -2376,6 +2376,9 @@ static void dce6_program_watermarks(struct radeon_device *rdev, · 2375 unmodified lines | ||
| 2376 | 2376 | c.full = dfixed_div(c, a); |
| 2377 | 2377 | priority_b_mark = dfixed_trunc(c); |
| 2378 | 2378 | priority_b_cnt |= priority_b_mark & PRIORITY_MARK_MASK; |
| 2379 | + | |
| 2380 | + /* Save number of lines the linebuffer leads before the scanout */ | |
| 2381 | + radeon_crtc->lb_vblank_lead_lines = DIV_ROUND_UP(lb_size, mode->crtc_hdisplay); | |
| 2379 | 2382 | } |
| 2380 | 2383 | |
| 2381 | 2384 | /* select wm A */ |
| @@ -6848,7 +6851,7 @@ restart_ih: · 4466 unmodified lines | ||
| 6848 | 6851 | if (queue_dp) |
| 6849 | 6852 | schedule_work(&rdev->dp_work); |
| 6850 | 6853 | if (queue_hotplug) |
| 6851 | − schedule_work(&rdev->hotplug_work); | |
| 6854 | + schedule_delayed_work(&rdev->hotplug_work, 0); | |
| 6852 | 6855 | if (queue_thermal && rdev->pm.dpm_enabled) |
| 6853 | 6856 | schedule_work(&rdev->pm.dpm.thermal.work); |
| 6854 | 6857 | rdev->ih.rptr = rptr; |
| @@ -67,6 +67,7 @@ static int rockchip_drm_gem_object_mmap(struct drm_gem_object *obj, · 66 unmodified lines | ||
| 67 | 67 | * VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap(). |
| 68 | 68 | */ |
| 69 | 69 | vma->vm_flags &= ~VM_PFNMAP; |
| 70 | + vma->vm_pgoff = 0; | |
| 70 | 71 | |
| 71 | 72 | ret = dma_mmap_attrs(drm->dev, vma, rk_obj->kvaddr, rk_obj->dma_addr, |
| 72 | 73 | obj->size, &rk_obj->dma_attrs); |
| @@ -374,6 +374,7 @@ static const struct of_device_id vop_driver_dt_match[] = { · 373 unmodified lines | ||
| 374 | 374 | .data = &rk3288_vop }, |
| 375 | 375 | {}, |
| 376 | 376 | }; |
| 377 | +MODULE_DEVICE_TABLE(of, vop_driver_dt_match); | |
| 377 | 378 | |
| 378 | 379 | static inline void vop_writel(struct vop *vop, uint32_t offset, uint32_t v) |
| 379 | 380 | { |
| @@ -959,8 +960,8 @@ static int vop_update_plane_event(struct drm_plane *plane, · 579 unmodified lines | ||
| 959 | 960 | val = (dest.y2 - dest.y1 - 1) << 16; |
| 960 | 961 | val |= (dest.x2 - dest.x1 - 1) & 0xffff; |
| 961 | 962 | VOP_WIN_SET(vop, win, dsp_info, val); |
| 962 | − val = (dsp_sty - 1) << 16; | |
| 963 | − val |= (dsp_stx - 1) & 0xffff; | |
| 963 | + val = dsp_sty << 16; | |
| 964 | + val |= dsp_stx & 0xffff; | |
| 964 | 965 | VOP_WIN_SET(vop, win, dsp_st, val); |
| 965 | 966 | VOP_WIN_SET(vop, win, rb_swap, rb_swap); |
| 966 | 967 | |
| @@ -1289,7 +1290,7 @@ static void vop_win_state_complete(struct vop_win *vop_win, · 322 unmodified lines | ||
| 1289 | 1290 | |
| 1290 | 1291 | if (state->event) { |
| 1291 | 1292 | spin_lock_irqsave(&drm->event_lock, flags); |
| 1292 | − drm_send_vblank_event(drm, -1, state->event); | |
| 1293 | + drm_crtc_send_vblank_event(crtc, state->event); | |
| 1293 | 1294 | spin_unlock_irqrestore(&drm->event_lock, flags); |
| 1294 | 1295 | } |
| 1295 | 1296 | |
| @@ -1575,32 +1576,25 @@ static int vop_initial(struct vop *vop) · 279 unmodified lines | ||
| 1575 | 1576 | return PTR_ERR(vop->dclk); |
| 1576 | 1577 | } |
| 1577 | 1578 | |
| 1578 | − ret = clk_prepare(vop->hclk); | |
| 1579 | − if (ret < 0) { | |
| 1580 | − dev_err(vop->dev, "failed to prepare hclk\n"); | |
| 1581 | − return ret; | |
| 1582 | − } | |
| 1583 | − | |
| 1584 | 1579 | ret = clk_prepare(vop->dclk); |
| 1585 | 1580 | if (ret < 0) { |
| 1586 | 1581 | dev_err(vop->dev, "failed to prepare dclk\n"); |
| 1587 | − goto err_unprepare_hclk; | |
| 1582 | + return ret; | |
| 1588 | 1583 | } |
| 1589 | 1584 | |
| 1590 | − ret = clk_prepare(vop->aclk); | |
| 1585 | + /* Enable both the hclk and aclk to setup the vop */ | |
| 1586 | + ret = clk_prepare_enable(vop->hclk); | |
| 1591 | 1587 | if (ret < 0) { |
| 1592 | − dev_err(vop->dev, "failed to prepare aclk\n"); | |
| 1588 | + dev_err(vop->dev, "failed to prepare/enable hclk\n"); | |
| 1593 | 1589 | goto err_unprepare_dclk; |
| 1594 | 1590 | } |
| 1595 | 1591 | |
| 1596 | − /* | |
| 1597 | − * enable hclk, so that we can config vop register. | |
| 1598 | − */ | |
| 1599 | − ret = clk_enable(vop->hclk); | |
| 1592 | + ret = clk_prepare_enable(vop->aclk); | |
| 1600 | 1593 | if (ret < 0) { |
| 1601 | − dev_err(vop->dev, "failed to prepare aclk\n"); | |
| 1602 | − goto err_unprepare_aclk; | |
| 1594 | + dev_err(vop->dev, "failed to prepare/enable aclk\n"); | |
| 1595 | + goto err_disable_hclk; | |
| 1603 | 1596 | } |
| 1597 | + | |
| 1604 | 1598 | /* |
| 1605 | 1599 | * do hclk_reset, reset all vop registers. |
| 1606 | 1600 | */ |
| @@ -1608,7 +1602,7 @@ static int vop_initial(struct vop *vop) · 1 unmodified line | ||
| 1608 | 1602 | if (IS_ERR(ahb_rst)) { |
| 1609 | 1603 | dev_err(vop->dev, "failed to get ahb reset\n"); |
| 1610 | 1604 | ret = PTR_ERR(ahb_rst); |
| 1611 | − goto err_disable_hclk; | |
| 1605 | + goto err_disable_aclk; | |
| 1612 | 1606 | } |
| 1613 | 1607 | reset_control_assert(ahb_rst); |
| 1614 | 1608 | usleep_range(10, 20); |
| @@ -1634,26 +1628,25 @@ static int vop_initial(struct vop *vop) · 19 unmodified lines | ||
| 1634 | 1628 | if (IS_ERR(vop->dclk_rst)) { |
| 1635 | 1629 | dev_err(vop->dev, "failed to get dclk reset\n"); |
| 1636 | 1630 | ret = PTR_ERR(vop->dclk_rst); |
| 1637 | − goto err_unprepare_aclk; | |
| 1631 | + goto err_disable_aclk; | |
| 1638 | 1632 | } |
| 1639 | 1633 | reset_control_assert(vop->dclk_rst); |
| 1640 | 1634 | usleep_range(10, 20); |
| 1641 | 1635 | reset_control_deassert(vop->dclk_rst); |
| 1642 | 1636 | |
| 1643 | 1637 | clk_disable(vop->hclk); |
| 1638 | + clk_disable(vop->aclk); | |
| 1644 | 1639 | |
| 1645 | 1640 | vop->is_enabled = false; |
| 1646 | 1641 | |
| 1647 | 1642 | return 0; |
| 1648 | 1643 | |
| 1644 | +err_disable_aclk: | |
| 1645 | + clk_disable_unprepare(vop->aclk); | |
| 1649 | 1646 | err_disable_hclk: |
| 1650 | − clk_disable(vop->hclk); | |
| 1651 | −err_unprepare_aclk: | |
| 1652 | − clk_unprepare(vop->aclk); | |
| 1647 | + clk_disable_unprepare(vop->hclk); | |
| 1653 | 1648 | err_unprepare_dclk: |
| 1654 | 1649 | clk_unprepare(vop->dclk); |
| 1655 | −err_unprepare_hclk: | |
| 1656 | − clk_unprepare(vop->hclk); | |
| 1657 | 1650 | return ret; |
| 1658 | 1651 | } |
| 1659 | 1652 | |
| @@ -180,7 +180,7 @@ int ttm_write_lock(struct ttm_lock *lock, bool interruptible) · 179 unmodified lines | ||
| 180 | 180 | spin_unlock(&lock->lock); |
| 181 | 181 | } |
| 182 | 182 | } else |
| 183 | − wait_event(lock->queue, __ttm_read_lock(lock)); | |
| 183 | + wait_event(lock->queue, __ttm_write_lock(lock)); | |
| 184 | 184 | |
| 185 | 185 | return ret; |
| 186 | 186 | } |
| @@ -412,7 +412,7 @@ static const struct drm_connector_funcs virtio_gpu_connector_funcs = { · 411 unmodified lines | ||
| 412 | 412 | .save = virtio_gpu_conn_save, |
| 413 | 413 | .restore = virtio_gpu_conn_restore, |
| 414 | 414 | .detect = virtio_gpu_conn_detect, |
| 415 | − .fill_modes = drm_helper_probe_single_connector_modes, | |
| 415 | + .fill_modes = drm_helper_probe_single_connector_modes_nomerge, | |
| 416 | 416 | .destroy = virtio_gpu_conn_destroy, |
| 417 | 417 | .reset = drm_atomic_helper_connector_reset, |
| 418 | 418 | .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state, |
| @@ -1233,6 +1233,7 @@ static void vmw_master_drop(struct drm_device *dev, · 1232 unmodified lines | ||
| 1233 | 1233 | |
| 1234 | 1234 | vmw_fp->locked_master = drm_master_get(file_priv->master); |
| 1235 | 1235 | ret = ttm_vt_lock(&vmaster->lock, false, vmw_fp->tfile); |
| 1236 | + vmw_kms_legacy_hotspot_clear(dev_priv); | |
| 1236 | 1237 | if (unlikely((ret != 0))) { |
| 1237 | 1238 | DRM_ERROR("Unable to lock TTM at VT switch.\n"); |
| 1238 | 1239 | drm_master_put(&vmw_fp->locked_master); |
| @@ -925,6 +925,7 @@ int vmw_kms_present(struct vmw_private *dev_priv, · 924 unmodified lines | ||
| 925 | 925 | uint32_t num_clips); |
| 926 | 926 | int vmw_kms_update_layout_ioctl(struct drm_device *dev, void *data, |
| 927 | 927 | struct drm_file *file_priv); |
| 928 | +void vmw_kms_legacy_hotspot_clear(struct vmw_private *dev_priv); | |
| 928 | 929 | |
| 929 | 930 | int vmw_dumb_create(struct drm_file *file_priv, |
| 930 | 931 | struct drm_device *dev, |
| @@ -390,7 +390,7 @@ void *vmw_fifo_reserve_dx(struct vmw_private *dev_priv, uint32_t bytes, · 389 unmodified lines | ||
| 390 | 390 | else if (ctx_id == SVGA3D_INVALID_ID) |
| 391 | 391 | ret = vmw_local_fifo_reserve(dev_priv, bytes); |
| 392 | 392 | else { |
| 393 | − WARN_ON("Command buffer has not been allocated.\n"); | |
| 393 | + WARN(1, "Command buffer has not been allocated.\n"); | |
| 394 | 394 | ret = NULL; |
| 395 | 395 | } |
| 396 | 396 | if (IS_ERR_OR_NULL(ret)) { |
| @@ -133,13 +133,19 @@ void vmw_cursor_update_position(struct vmw_private *dev_priv, · 132 unmodified lines | ||
| 133 | 133 | vmw_mmio_write(++count, fifo_mem + SVGA_FIFO_CURSOR_COUNT); |
| 134 | 134 | } |
| 135 | 135 | |
| 136 | −int vmw_du_crtc_cursor_set(struct drm_crtc *crtc, struct drm_file *file_priv, | |
| 137 | − uint32_t handle, uint32_t width, uint32_t height) | |
| 136 | + | |
| 137 | +/* | |
| 138 | + * vmw_du_crtc_cursor_set2 - Driver cursor_set2 callback. | |
| 139 | + */ | |
| 140 | +int vmw_du_crtc_cursor_set2(struct drm_crtc *crtc, struct drm_file *file_priv, | |
| 141 | + uint32_t handle, uint32_t width, uint32_t height, | |
| 142 | + int32_t hot_x, int32_t hot_y) | |
| 138 | 143 | { |
| 139 | 144 | struct vmw_private *dev_priv = vmw_priv(crtc->dev); |
| 140 | 145 | struct vmw_display_unit *du = vmw_crtc_to_du(crtc); |
| 141 | 146 | struct vmw_surface *surface = NULL; |
| 142 | 147 | struct vmw_dma_buffer *dmabuf = NULL; |
| 148 | + s32 hotspot_x, hotspot_y; | |
| 143 | 149 | int ret; |
| 144 | 150 | |
| 145 | 151 | /* |
| @@ -151,6 +157,8 @@ int vmw_du_crtc_cursor_set(struct drm_crtc *crtc, struct drm_file *file_priv, · 5 unmodified lines | ||
| 151 | 157 | */ |
| 152 | 158 | drm_modeset_unlock_crtc(crtc); |
| 153 | 159 | drm_modeset_lock_all(dev_priv->dev); |
| 160 | + hotspot_x = hot_x + du->hotspot_x; | |
| 161 | + hotspot_y = hot_y + du->hotspot_y; | |
| 154 | 162 | |
| 155 | 163 | /* A lot of the code assumes this */ |
| 156 | 164 | if (handle && (width != 64 || height != 64)) { |
| @@ -187,31 +195,34 @@ int vmw_du_crtc_cursor_set(struct drm_crtc *crtc, struct drm_file *file_priv, · 30 unmodified lines | ||
| 187 | 195 | vmw_dmabuf_unreference(&du->cursor_dmabuf); |
| 188 | 196 | |
| 189 | 197 | /* setup new image */ |
| 198 | + ret = 0; | |
| 190 | 199 | if (surface) { |
| 191 | 200 | /* vmw_user_surface_lookup takes one reference */ |
| 192 | 201 | du->cursor_surface = surface; |
| 193 | 202 | |
| 194 | 203 | du->cursor_surface->snooper.crtc = crtc; |
| 195 | 204 | du->cursor_age = du->cursor_surface->snooper.age; |
| 196 | − vmw_cursor_update_image(dev_priv, surface->snooper.image, | |
| 197 | − 64, 64, du->hotspot_x, du->hotspot_y); | |
| 205 | + ret = vmw_cursor_update_image(dev_priv, surface->snooper.image, | |
| 206 | + 64, 64, hotspot_x, hotspot_y); | |
| 198 | 207 | } else if (dmabuf) { |
| 199 | 208 | /* vmw_user_surface_lookup takes one reference */ |
| 200 | 209 | du->cursor_dmabuf = dmabuf; |
| 201 | 210 | |
| 202 | 211 | ret = vmw_cursor_update_dmabuf(dev_priv, dmabuf, width, height, |
| 203 | − du->hotspot_x, du->hotspot_y); | |
| 212 | + hotspot_x, hotspot_y); | |
| 204 | 213 | } else { |
| 205 | 214 | vmw_cursor_update_position(dev_priv, false, 0, 0); |
| 206 | − ret = 0; | |
| 207 | 215 | goto out; |
| 208 | 216 | } |
| 209 | 217 | |
| 210 | − vmw_cursor_update_position(dev_priv, true, | |
| 211 | − du->cursor_x + du->hotspot_x, | |
| 212 | − du->cursor_y + du->hotspot_y); | |
| 218 | + if (!ret) { | |
| 219 | + vmw_cursor_update_position(dev_priv, true, | |
| 220 | + du->cursor_x + hotspot_x, | |
| 221 | + du->cursor_y + hotspot_y); | |
| 222 | + du->core_hotspot_x = hot_x; | |
| 223 | + du->core_hotspot_y = hot_y; | |
| 224 | + } | |
| 213 | 225 | |
| 214 | − ret = 0; | |
| 215 | 226 | out: |
| 216 | 227 | drm_modeset_unlock_all(dev_priv->dev); |
| 217 | 228 | drm_modeset_lock_crtc(crtc, crtc->cursor); |
| @@ -239,8 +250,10 @@ int vmw_du_crtc_cursor_move(struct drm_crtc *crtc, int x, int y) · 21 unmodified lines | ||
| 239 | 250 | drm_modeset_lock_all(dev_priv->dev); |
| 240 | 251 | |
| 241 | 252 | vmw_cursor_update_position(dev_priv, shown, |
| 242 | − du->cursor_x + du->hotspot_x, | |
| 243 | − du->cursor_y + du->hotspot_y); | |
| 253 | + du->cursor_x + du->hotspot_x + | |
| 254 | + du->core_hotspot_x, | |
| 255 | + du->cursor_y + du->hotspot_y + | |
| 256 | + du->core_hotspot_y); | |
| 244 | 257 | |
| 245 | 258 | drm_modeset_unlock_all(dev_priv->dev); |
| 246 | 259 | drm_modeset_lock_crtc(crtc, crtc->cursor); |
| @@ -334,6 +347,29 @@ err_unreserve: · 87 unmodified lines | ||
| 334 | 347 | ttm_bo_unreserve(bo); |
| 335 | 348 | } |
| 336 | 349 | |
| 350 | +/** | |
| 351 | + * vmw_kms_legacy_hotspot_clear - Clear legacy hotspots | |
| 352 | + * | |
| 353 | + * @dev_priv: Pointer to the device private struct. | |
| 354 | + * | |
| 355 | + * Clears all legacy hotspots. | |
| 356 | + */ | |
| 357 | +void vmw_kms_legacy_hotspot_clear(struct vmw_private *dev_priv) | |
| 358 | +{ | |
| 359 | + struct drm_device *dev = dev_priv->dev; | |
| 360 | + struct vmw_display_unit *du; | |
| 361 | + struct drm_crtc *crtc; | |
| 362 | + | |
| 363 | + drm_modeset_lock_all(dev); | |
| 364 | + drm_for_each_crtc(crtc, dev) { | |
| 365 | + du = vmw_crtc_to_du(crtc); | |
| 366 | + | |
| 367 | + du->hotspot_x = 0; | |
| 368 | + du->hotspot_y = 0; | |
| 369 | + } | |
| 370 | + drm_modeset_unlock_all(dev); | |
| 371 | +} | |
| 372 | + | |
| 337 | 373 | void vmw_kms_cursor_post_execbuf(struct vmw_private *dev_priv) |
| 338 | 374 | { |
| 339 | 375 | struct drm_device *dev = dev_priv->dev; |
| @@ -351,7 +387,9 @@ void vmw_kms_cursor_post_execbuf(struct vmw_private *dev_priv) · 11 unmodified lines | ||
| 351 | 387 | du->cursor_age = du->cursor_surface->snooper.age; |
| 352 | 388 | vmw_cursor_update_image(dev_priv, |
| 353 | 389 | du->cursor_surface->snooper.image, |
| 354 | − 64, 64, du->hotspot_x, du->hotspot_y); | |
| 390 | + 64, 64, | |
| 391 | + du->hotspot_x + du->core_hotspot_x, | |
| 392 | + du->hotspot_y + du->core_hotspot_y); | |
| 355 | 393 | } |
| 356 | 394 | |
| 357 | 395 | mutex_unlock(&dev->mode_config.mutex); |
| @@ -159,6 +159,8 @@ struct vmw_display_unit { · 158 unmodified lines | ||
| 159 | 159 | |
| 160 | 160 | int hotspot_x; |
| 161 | 161 | int hotspot_y; |
| 162 | + s32 core_hotspot_x; | |
| 163 | + s32 core_hotspot_y; | |
| 162 | 164 | |
| 163 | 165 | unsigned unit; |
| 164 | 166 | |
| @@ -193,8 +195,9 @@ void vmw_du_crtc_restore(struct drm_crtc *crtc); · 28 unmodified lines | ||
| 193 | 195 | void vmw_du_crtc_gamma_set(struct drm_crtc *crtc, |
| 194 | 196 | u16 *r, u16 *g, u16 *b, |
| 195 | 197 | uint32_t start, uint32_t size); |
| 196 | −int vmw_du_crtc_cursor_set(struct drm_crtc *crtc, struct drm_file *file_priv, | |
| 197 | − uint32_t handle, uint32_t width, uint32_t height); | |
| 198 | +int vmw_du_crtc_cursor_set2(struct drm_crtc *crtc, struct drm_file *file_priv, | |
| 199 | + uint32_t handle, uint32_t width, uint32_t height, | |
| 200 | + int32_t hot_x, int32_t hot_y); | |
| 198 | 201 | int vmw_du_crtc_cursor_move(struct drm_crtc *crtc, int x, int y); |
| 199 | 202 | int vmw_du_connector_dpms(struct drm_connector *connector, int mode); |
| 200 | 203 | void vmw_du_connector_save(struct drm_connector *connector); |
| @@ -297,7 +297,7 @@ static int vmw_ldu_crtc_set_config(struct drm_mode_set *set) · 296 unmodified lines | ||
| 297 | 297 | static struct drm_crtc_funcs vmw_legacy_crtc_funcs = { |
| 298 | 298 | .save = vmw_du_crtc_save, |
| 299 | 299 | .restore = vmw_du_crtc_restore, |
| 300 | − .cursor_set = vmw_du_crtc_cursor_set, | |
| 300 | + .cursor_set2 = vmw_du_crtc_cursor_set2, | |
| 301 | 301 | .cursor_move = vmw_du_crtc_cursor_move, |
| 302 | 302 | .gamma_set = vmw_du_crtc_gamma_set, |
| 303 | 303 | .destroy = vmw_ldu_crtc_destroy, |
| @@ -533,7 +533,7 @@ out_no_fence: · 532 unmodified lines | ||
| 533 | 533 | static struct drm_crtc_funcs vmw_screen_object_crtc_funcs = { |
| 534 | 534 | .save = vmw_du_crtc_save, |
| 535 | 535 | .restore = vmw_du_crtc_restore, |
| 536 | − .cursor_set = vmw_du_crtc_cursor_set, | |
| 536 | + .cursor_set2 = vmw_du_crtc_cursor_set2, | |
| 537 | 537 | .cursor_move = vmw_du_crtc_cursor_move, |
| 538 | 538 | .gamma_set = vmw_du_crtc_gamma_set, |
| 539 | 539 | .destroy = vmw_sou_crtc_destroy, |
| @@ -1043,7 +1043,7 @@ out_finish: · 1042 unmodified lines | ||
| 1043 | 1043 | static struct drm_crtc_funcs vmw_stdu_crtc_funcs = { |
| 1044 | 1044 | .save = vmw_du_crtc_save, |
| 1045 | 1045 | .restore = vmw_du_crtc_restore, |
| 1046 | − .cursor_set = vmw_du_crtc_cursor_set, | |
| 1046 | + .cursor_set2 = vmw_du_crtc_cursor_set2, | |
| 1047 | 1047 | .cursor_move = vmw_du_crtc_cursor_move, |
| 1048 | 1048 | .gamma_set = vmw_du_crtc_gamma_set, |
| 1049 | 1049 | .destroy = vmw_stdu_crtc_destroy, |
| @@ -28,6 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <linux/irqchip/chained_irq.h> |
| 29 | 29 | #include <linux/irqdomain.h> |
| 30 | 30 | #include <linux/of_device.h> |
| 31 | +#include <linux/of_graph.h> | |
| 31 | 32 | |
| 32 | 33 | #include <drm/drm_fourcc.h> |
| 33 | 34 | |
| @@ -993,11 +994,25 @@ static void platform_device_unregister_children(struct platform_device *pdev) · 959 unmodified lines | ||
| 993 | 994 | struct ipu_platform_reg { |
| 994 | 995 | struct ipu_client_platformdata pdata; |
| 995 | 996 | const char *name; |
| 996 | − int reg_offset; | |
| 997 | 997 | }; |
| 998 | 998 | |
| 999 | +/* These must be in the order of the corresponding device tree port nodes */ | |
| 999 | 1000 | static const struct ipu_platform_reg client_reg[] = { |
| 1000 | 1001 | { |
| 1002 | + .pdata = { | |
| 1003 | + .csi = 0, | |
| 1004 | + .dma[0] = IPUV3_CHANNEL_CSI0, | |
| 1005 | + .dma[1] = -EINVAL, | |
| 1006 | + }, | |
| 1007 | + .name = "imx-ipuv3-camera", | |
| 1008 | + }, { | |
| 1009 | + .pdata = { | |
| 1010 | + .csi = 1, | |
| 1011 | + .dma[0] = IPUV3_CHANNEL_CSI1, | |
| 1012 | + .dma[1] = -EINVAL, | |
| 1013 | + }, | |
| 1014 | + .name = "imx-ipuv3-camera", | |
| 1015 | + }, { | |
| 1001 | 1016 | .pdata = { |
| 1002 | 1017 | .di = 0, |
| 1003 | 1018 | .dc = 5, |
| @@ -1015,22 +1030,6 @@ static const struct ipu_platform_reg client_reg[] = { · 11 unmodified lines | ||
| 1015 | 1030 | .dma[1] = -EINVAL, |
| 1016 | 1031 | }, |
| 1017 | 1032 | .name = "imx-ipuv3-crtc", |
| 1018 | − }, { | |
| 1019 | − .pdata = { | |
| 1020 | − .csi = 0, | |
| 1021 | − .dma[0] = IPUV3_CHANNEL_CSI0, | |
| 1022 | − .dma[1] = -EINVAL, | |
| 1023 | − }, | |
| 1024 | − .reg_offset = IPU_CM_CSI0_REG_OFS, | |
| 1025 | − .name = "imx-ipuv3-camera", | |
| 1026 | − }, { | |
| 1027 | − .pdata = { | |
| 1028 | − .csi = 1, | |
| 1029 | − .dma[0] = IPUV3_CHANNEL_CSI1, | |
| 1030 | − .dma[1] = -EINVAL, | |
| 1031 | − }, | |
| 1032 | − .reg_offset = IPU_CM_CSI1_REG_OFS, | |
| 1033 | − .name = "imx-ipuv3-camera", | |
| 1034 | 1033 | }, |
| 1035 | 1034 | }; |
| 1036 | 1035 | |
| @@ -1051,22 +1050,30 @@ static int ipu_add_client_devices(struct ipu_soc *ipu, unsigned long ipu_base) · 14 unmodified lines | ||
| 1051 | 1050 | for (i = 0; i < ARRAY_SIZE(client_reg); i++) { |
| 1052 | 1051 | const struct ipu_platform_reg *reg = &client_reg[i]; |
| 1053 | 1052 | struct platform_device *pdev; |
| 1054 | − struct resource res; | |
| 1055 | − | |
| 1056 | − if (reg->reg_offset) { | |
| 1057 | − memset(&res, 0, sizeof(res)); | |
| 1058 | − res.flags = IORESOURCE_MEM; | |
| 1059 | − res.start = ipu_base + ipu->devtype->cm_ofs + reg->reg_offset; | |
| 1060 | − res.end = res.start + PAGE_SIZE - 1; | |
| 1061 | − pdev = platform_device_register_resndata(dev, reg->name, | |
| 1062 | − id++, &res, 1, ®->pdata, sizeof(reg->pdata)); | |
| 1063 | − } else { | |
| 1064 | − pdev = platform_device_register_data(dev, reg->name, | |
| 1065 | − id++, ®->pdata, sizeof(reg->pdata)); | |
| 1053 | + | |
| 1054 | + pdev = platform_device_alloc(reg->name, id++); | |
| 1055 | + if (!pdev) { | |
| 1056 | + ret = -ENOMEM; | |
| 1057 | + goto err_register; | |
| 1058 | + } | |
| 1059 | + | |
| 1060 | + pdev->dev.parent = dev; | |
| 1061 | + | |
| 1062 | + /* Associate subdevice with the corresponding port node */ | |
| 1063 | + pdev->dev.of_node = of_graph_get_port_by_id(dev->of_node, i); | |
| 1064 | + if (!pdev->dev.of_node) { | |
| 1065 | + dev_err(dev, "missing port@%d node in %s\n", i, | |
| 1066 | + dev->of_node->full_name); | |
| 1067 | + ret = -ENODEV; | |
| 1068 | + goto err_register; | |
| 1066 | 1069 | } |
| 1067 | 1070 | |
| 1068 | − if (IS_ERR(pdev)) { | |
| 1069 | − ret = PTR_ERR(pdev); | |
| 1071 | + ret = platform_device_add_data(pdev, ®->pdata, | |
| 1072 | + sizeof(reg->pdata)); | |
| 1073 | + if (!ret) | |
| 1074 | + ret = platform_device_add(pdev); | |
| 1075 | + if (ret) { | |
| 1076 | + platform_device_put(pdev); | |
| 1070 | 1077 | goto err_register; |
| 1071 | 1078 | } |
| 1072 | 1079 | } |
| @@ -395,8 +395,10 @@ int vga_get(struct pci_dev *pdev, unsigned int rsrc, int interruptible) · 394 unmodified lines | ||
| 395 | 395 | set_current_state(interruptible ? |
| 396 | 396 | TASK_INTERRUPTIBLE : |
| 397 | 397 | TASK_UNINTERRUPTIBLE); |
| 398 | − if (signal_pending(current)) { | |
| 399 | − rc = -EINTR; | |
| 398 | + if (interruptible && signal_pending(current)) { | |
| 399 | + __set_current_state(TASK_RUNNING); | |
| 400 | + remove_wait_queue(&vga_wait_queue, &wait); | |
| 401 | + rc = -ERESTARTSYS; | |
| 400 | 402 | break; |
| 401 | 403 | } |
| 402 | 404 | schedule(); |
| @@ -316,11 +316,6 @@ · 315 unmodified lines | ||
| 316 | 316 | #define USB_DEVICE_ID_DWAV_EGALAX_MULTITOUCH_A001 0xa001 |
| 317 | 317 | |
| 318 | 318 | #define USB_VENDOR_ID_ELAN 0x04f3 |
| 319 | −#define USB_DEVICE_ID_ELAN_TOUCHSCREEN 0x0089 | |
| 320 | −#define USB_DEVICE_ID_ELAN_TOUCHSCREEN_009B 0x009b | |
| 321 | −#define USB_DEVICE_ID_ELAN_TOUCHSCREEN_0103 0x0103 | |
| 322 | −#define USB_DEVICE_ID_ELAN_TOUCHSCREEN_010c 0x010c | |
| 323 | −#define USB_DEVICE_ID_ELAN_TOUCHSCREEN_016F 0x016f | |
| 324 | 319 | |
| 325 | 320 | #define USB_VENDOR_ID_ELECOM 0x056e |
| 326 | 321 | #define USB_DEVICE_ID_ELECOM_BM084 0x0061 |
| @@ -609,6 +604,7 @@ · 282 unmodified lines | ||
| 609 | 604 | #define USB_DEVICE_ID_LOGITECH_HARMONY_FIRST 0xc110 |
| 610 | 605 | #define USB_DEVICE_ID_LOGITECH_HARMONY_LAST 0xc14f |
| 611 | 606 | #define USB_DEVICE_ID_LOGITECH_HARMONY_PS3 0x0306 |
| 607 | +#define USB_DEVICE_ID_LOGITECH_KEYBOARD_G710_PLUS 0xc24d | |
| 612 | 608 | #define USB_DEVICE_ID_LOGITECH_MOUSE_C01A 0xc01a |
| 613 | 609 | #define USB_DEVICE_ID_LOGITECH_MOUSE_C05A 0xc05a |
| 614 | 610 | #define USB_DEVICE_ID_LOGITECH_MOUSE_C06A 0xc06a |
| @@ -665,8 +665,9 @@ static int lg_probe(struct hid_device *hdev, const struct hid_device_id *id) · 664 unmodified lines | ||
| 665 | 665 | struct lg_drv_data *drv_data; |
| 666 | 666 | int ret; |
| 667 | 667 | |
| 668 | − /* Only work with the 1st interface (G29 presents multiple) */ | |
| 669 | − if (iface_num != 0) { | |
| 668 | + /* G29 only work with the 1st interface */ | |
| 669 | + if ((hdev->product == USB_DEVICE_ID_LOGITECH_G29_WHEEL) && | |
| 670 | + (iface_num != 0)) { | |
| 670 | 671 | dbg_hid("%s: ignoring ifnum %d\n", __func__, iface_num); |
| 671 | 672 | return -ENODEV; |
| 672 | 673 | } |
| @@ -72,11 +72,7 @@ static const struct hid_blacklist { · 71 unmodified lines | ||
| 72 | 72 | { USB_VENDOR_ID_CHICONY, USB_DEVICE_ID_CHICONY_PIXART_USB_OPTICAL_MOUSE, HID_QUIRK_ALWAYS_POLL }, |
| 73 | 73 | { USB_VENDOR_ID_DMI, USB_DEVICE_ID_DMI_ENC, HID_QUIRK_NOGET }, |
| 74 | 74 | { USB_VENDOR_ID_DRAGONRISE, USB_DEVICE_ID_DRAGONRISE_WIIU, HID_QUIRK_MULTI_INPUT }, |
| 75 | − { USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ELAN_TOUCHSCREEN, HID_QUIRK_ALWAYS_POLL }, | |
| 76 | − { USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ELAN_TOUCHSCREEN_009B, HID_QUIRK_ALWAYS_POLL }, | |
| 77 | − { USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ELAN_TOUCHSCREEN_0103, HID_QUIRK_ALWAYS_POLL }, | |
| 78 | − { USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ELAN_TOUCHSCREEN_010c, HID_QUIRK_ALWAYS_POLL }, | |
| 79 | − { USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ELAN_TOUCHSCREEN_016F, HID_QUIRK_ALWAYS_POLL }, | |
| 75 | + { USB_VENDOR_ID_ELAN, HID_ANY_ID, HID_QUIRK_ALWAYS_POLL }, | |
| 80 | 76 | { USB_VENDOR_ID_ELO, USB_DEVICE_ID_ELO_TS2700, HID_QUIRK_NOGET }, |
| 81 | 77 | { USB_VENDOR_ID_FORMOSA, USB_DEVICE_ID_FORMOSA_IR_RECEIVER, HID_QUIRK_NO_INIT_REPORTS }, |
| 82 | 78 | { USB_VENDOR_ID_FREESCALE, USB_DEVICE_ID_FREESCALE_MX28, HID_QUIRK_NOGET }, |
| @@ -84,6 +80,7 @@ static const struct hid_blacklist { · 1 unmodified line | ||
| 84 | 80 | { USB_VENDOR_ID_HP, USB_PRODUCT_ID_HP_LOGITECH_OEM_USB_OPTICAL_MOUSE_0B4A, HID_QUIRK_ALWAYS_POLL }, |
| 85 | 81 | { USB_VENDOR_ID_HP, USB_PRODUCT_ID_HP_PIXART_OEM_USB_OPTICAL_MOUSE, HID_QUIRK_ALWAYS_POLL }, |
| 86 | 82 | { USB_VENDOR_ID_LOGITECH, USB_DEVICE_ID_LOGITECH_C077, HID_QUIRK_ALWAYS_POLL }, |
| 83 | + { USB_VENDOR_ID_LOGITECH, USB_DEVICE_ID_LOGITECH_KEYBOARD_G710_PLUS, HID_QUIRK_NOGET }, | |
| 87 | 84 | { USB_VENDOR_ID_LOGITECH, USB_DEVICE_ID_LOGITECH_MOUSE_C01A, HID_QUIRK_ALWAYS_POLL }, |
| 88 | 85 | { USB_VENDOR_ID_LOGITECH, USB_DEVICE_ID_LOGITECH_MOUSE_C05A, HID_QUIRK_ALWAYS_POLL }, |
| 89 | 86 | { USB_VENDOR_ID_LOGITECH, USB_DEVICE_ID_LOGITECH_MOUSE_C06A, HID_QUIRK_ALWAYS_POLL }, |
| @@ -339,7 +336,8 @@ static const struct hid_blacklist *usbhid_exists_squirk(const u16 idVendor, · 249 unmodified lines | ||
| 339 | 336 | |
| 340 | 337 | for (; hid_blacklist[n].idVendor; n++) |
| 341 | 338 | if (hid_blacklist[n].idVendor == idVendor && |
| 342 | − hid_blacklist[n].idProduct == idProduct) | |
| 339 | + (hid_blacklist[n].idProduct == (__u16) HID_ANY_ID || | |
| 340 | + hid_blacklist[n].idProduct == idProduct)) | |
| 343 | 341 | bl_entry = &hid_blacklist[n]; |
| 344 | 342 | |
| 345 | 343 | if (bl_entry != NULL) |
| @@ -839,8 +839,10 @@ static int vadc_get_dt_data(struct vadc_priv *vadc, struct device_node *node) · 838 unmodified lines | ||
| 839 | 839 | |
| 840 | 840 | for_each_available_child_of_node(node, child) { |
| 841 | 841 | ret = vadc_get_dt_channel_data(vadc->dev, &prop, child); |
| 842 | − if (ret) | |
| 842 | + if (ret) { | |
| 843 | + of_node_put(child); | |
| 843 | 844 | return ret; |
| 845 | + } | |
| 844 | 846 | |
| 845 | 847 | vadc->chan_props[index] = prop; |
| 846 | 848 | |
| @@ -302,7 +302,7 @@ static int iio_scan_mask_set(struct iio_dev *indio_dev, · 301 unmodified lines | ||
| 302 | 302 | if (trialmask == NULL) |
| 303 | 303 | return -ENOMEM; |
| 304 | 304 | if (!indio_dev->masklength) { |
| 305 | − WARN_ON("Trying to set scanmask prior to registering buffer\n"); | |
| 305 | + WARN(1, "Trying to set scanmask prior to registering buffer\n"); | |
| 306 | 306 | goto err_invalid_mask; |
| 307 | 307 | } |
| 308 | 308 | bitmap_copy(trialmask, buffer->scan_mask, indio_dev->masklength); |
| @@ -655,7 +655,7 @@ int __iio_device_attr_init(struct device_attribute *dev_attr, · 654 unmodified lines | ||
| 655 | 655 | break; |
| 656 | 656 | case IIO_SEPARATE: |
| 657 | 657 | if (!chan->indexed) { |
| 658 | − WARN_ON("Differential channels must be indexed\n"); | |
| 658 | + WARN(1, "Differential channels must be indexed\n"); | |
| 659 | 659 | ret = -EINVAL; |
| 660 | 660 | goto error_free_full_postfix; |
| 661 | 661 | } |
| @@ -453,6 +453,7 @@ static int apds9960_set_power_state(struct apds9960_data *data, bool on) · 452 unmodified lines | ||
| 453 | 453 | usleep_range(data->als_adc_int_us, |
| 454 | 454 | APDS9960_MAX_INT_TIME_IN_US); |
| 455 | 455 | } else { |
| 456 | + pm_runtime_mark_last_busy(dev); | |
| 456 | 457 | ret = pm_runtime_put_autosuspend(dev); |
| 457 | 458 | } |
| 458 | 459 | |
| @@ -130,10 +130,10 @@ static int lidar_get_measurement(struct lidar_data *data, u16 *reg) · 129 unmodified lines | ||
| 130 | 130 | if (ret < 0) |
| 131 | 131 | break; |
| 132 | 132 | |
| 133 | − /* return 0 since laser is likely pointed out of range */ | |
| 133 | + /* return -EINVAL since laser is likely pointed out of range */ | |
| 134 | 134 | if (ret & LIDAR_REG_STATUS_INVALID) { |
| 135 | 135 | *reg = 0; |
| 136 | − ret = 0; | |
| 136 | + ret = -EINVAL; | |
| 137 | 137 | break; |
| 138 | 138 | } |
| 139 | 139 | |
| @@ -197,7 +197,7 @@ static irqreturn_t lidar_trigger_handler(int irq, void *private) · 57 unmodified lines | ||
| 197 | 197 | if (!ret) { |
| 198 | 198 | iio_push_to_buffers_with_timestamp(indio_dev, data->buffer, |
| 199 | 199 | iio_get_time_ns()); |
| 200 | − } else { | |
| 200 | + } else if (ret != -EINVAL) { | |
| 201 | 201 | dev_err(&data->client->dev, "cannot read LIDAR measurement"); |
| 202 | 202 | } |
| 203 | 203 | |
| @@ -1126,10 +1126,7 @@ static bool validate_ipv4_net_dev(struct net_device *net_dev, · 1125 unmodified lines | ||
| 1126 | 1126 | |
| 1127 | 1127 | rcu_read_lock(); |
| 1128 | 1128 | err = fib_lookup(dev_net(net_dev), &fl4, &res, 0); |
| 1129 | − if (err) | |
| 1130 | − return false; | |
| 1131 | − | |
| 1132 | − ret = FIB_RES_DEV(res) == net_dev; | |
| 1129 | + ret = err == 0 && FIB_RES_DEV(res) == net_dev; | |
| 1133 | 1130 | rcu_read_unlock(); |
| 1134 | 1131 | |
| 1135 | 1132 | return ret; |
| @@ -1811,6 +1811,11 @@ static int validate_mad(const struct ib_mad_hdr *mad_hdr, · 1810 unmodified lines | ||
| 1811 | 1811 | if (qp_num == 0) |
| 1812 | 1812 | valid = 1; |
| 1813 | 1813 | } else { |
| 1814 | + /* CM attributes other than ClassPortInfo only use Send method */ | |
| 1815 | + if ((mad_hdr->mgmt_class == IB_MGMT_CLASS_CM) && | |
| 1816 | + (mad_hdr->attr_id != IB_MGMT_CLASSPORTINFO_ATTR_ID) && | |
| 1817 | + (mad_hdr->method != IB_MGMT_METHOD_SEND)) | |
| 1818 | + goto out; | |
| 1814 | 1819 | /* Filter GSI packets sent to QP0 */ |
| 1815 | 1820 | if (qp_num != 0) |
| 1816 | 1821 | valid = 1; |
| @@ -512,7 +512,7 @@ static int ib_nl_get_path_rec_attrs_len(ib_sa_comp_mask comp_mask) · 511 unmodified lines | ||
| 512 | 512 | return len; |
| 513 | 513 | } |
| 514 | 514 | |
| 515 | −static int ib_nl_send_msg(struct ib_sa_query *query) | |
| 515 | +static int ib_nl_send_msg(struct ib_sa_query *query, gfp_t gfp_mask) | |
| 516 | 516 | { |
| 517 | 517 | struct sk_buff *skb = NULL; |
| 518 | 518 | struct nlmsghdr *nlh; |
| @@ -526,7 +526,7 @@ static int ib_nl_send_msg(struct ib_sa_query *query) · 7 unmodified lines | ||
| 526 | 526 | if (len <= 0) |
| 527 | 527 | return -EMSGSIZE; |
| 528 | 528 | |
| 529 | − skb = nlmsg_new(len, GFP_KERNEL); | |
| 529 | + skb = nlmsg_new(len, gfp_mask); | |
| 530 | 530 | if (!skb) |
| 531 | 531 | return -ENOMEM; |
| 532 | 532 | |
| @@ -544,7 +544,7 @@ static int ib_nl_send_msg(struct ib_sa_query *query) · 11 unmodified lines | ||
| 544 | 544 | /* Repair the nlmsg header length */ |
| 545 | 545 | nlmsg_end(skb, nlh); |
| 546 | 546 | |
| 547 | − ret = ibnl_multicast(skb, nlh, RDMA_NL_GROUP_LS, GFP_KERNEL); | |
| 547 | + ret = ibnl_multicast(skb, nlh, RDMA_NL_GROUP_LS, gfp_mask); | |
| 548 | 548 | if (!ret) |
| 549 | 549 | ret = len; |
| 550 | 550 | else |
| @@ -553,7 +553,7 @@ static int ib_nl_send_msg(struct ib_sa_query *query) · 2 unmodified lines | ||
| 553 | 553 | return ret; |
| 554 | 554 | } |
| 555 | 555 | |
| 556 | −static int ib_nl_make_request(struct ib_sa_query *query) | |
| 556 | +static int ib_nl_make_request(struct ib_sa_query *query, gfp_t gfp_mask) | |
| 557 | 557 | { |
| 558 | 558 | unsigned long flags; |
| 559 | 559 | unsigned long delay; |
| @@ -562,25 +562,27 @@ static int ib_nl_make_request(struct ib_sa_query *query) · 2 unmodified lines | ||
| 562 | 562 | INIT_LIST_HEAD(&query->list); |
| 563 | 563 | query->seq = (u32)atomic_inc_return(&ib_nl_sa_request_seq); |
| 564 | 564 | |
| 565 | + /* Put the request on the list first.*/ | |
| 565 | 566 | spin_lock_irqsave(&ib_nl_request_lock, flags); |
| 566 | − ret = ib_nl_send_msg(query); | |
| 567 | − if (ret <= 0) { | |
| 568 | − ret = -EIO; | |
| 569 | − goto request_out; | |
| 570 | − } else { | |
| 571 | − ret = 0; | |
| 572 | − } | |
| 573 | − | |
| 574 | 567 | delay = msecs_to_jiffies(sa_local_svc_timeout_ms); |
| 575 | 568 | query->timeout = delay + jiffies; |
| 576 | 569 | list_add_tail(&query->list, &ib_nl_request_list); |
| 577 | 570 | /* Start the timeout if this is the only request */ |
| 578 | 571 | if (ib_nl_request_list.next == &query->list) |
| 579 | 572 | queue_delayed_work(ib_nl_wq, &ib_nl_timed_work, delay); |
| 580 | − | |
| 581 | −request_out: | |
| 582 | 573 | spin_unlock_irqrestore(&ib_nl_request_lock, flags); |
| 583 | 574 | |
| 575 | + ret = ib_nl_send_msg(query, gfp_mask); | |
| 576 | + if (ret <= 0) { | |
| 577 | + ret = -EIO; | |
| 578 | + /* Remove the request */ | |
| 579 | + spin_lock_irqsave(&ib_nl_request_lock, flags); | |
| 580 | + list_del(&query->list); | |
| 581 | + spin_unlock_irqrestore(&ib_nl_request_lock, flags); | |
| 582 | + } else { | |
| 583 | + ret = 0; | |
| 584 | + } | |
| 585 | + | |
| 584 | 586 | return ret; |
| 585 | 587 | } |
| 586 | 588 | |
| @@ -1108,7 +1110,7 @@ static int send_mad(struct ib_sa_query *query, int timeout_ms, gfp_t gfp_mask) · 521 unmodified lines | ||
| 1108 | 1110 | |
| 1109 | 1111 | if (query->flags & IB_SA_ENABLE_LOCAL_SERVICE) { |
| 1110 | 1112 | if (!ibnl_chk_listeners(RDMA_NL_GROUP_LS)) { |
| 1111 | − if (!ib_nl_make_request(query)) | |
| 1113 | + if (!ib_nl_make_request(query, gfp_mask)) | |
| 1112 | 1114 | return id; |
| 1113 | 1115 | } |
| 1114 | 1116 | ib_sa_disable_local_svc(query); |
| @@ -62,9 +62,11 @@ static struct uverbs_lock_class rule_lock_class = { .name = "RULE-uobj" }; · 61 unmodified lines | ||
| 62 | 62 | * The ib_uobject locking scheme is as follows: |
| 63 | 63 | * |
| 64 | 64 | * - ib_uverbs_idr_lock protects the uverbs idrs themselves, so it |
| 65 | − * needs to be held during all idr operations. When an object is | |
| 65 | + * needs to be held during all idr write operations. When an object is | |
| 66 | 66 | * looked up, a reference must be taken on the object's kref before |
| 67 | − * dropping this lock. | |
| 67 | + * dropping this lock. For read operations, the rcu_read_lock() | |
| 68 | + * and rcu_write_lock() but similarly the kref reference is grabbed | |
| 69 | + * before the rcu_read_unlock(). | |
| 68 | 70 | * |
| 69 | 71 | * - Each object also has an rwsem. This rwsem must be held for |
| 70 | 72 | * reading while an operation that uses the object is performed. |
| @@ -96,7 +98,7 @@ static void init_uobj(struct ib_uobject *uobj, u64 user_handle, · 25 unmodified lines | ||
| 96 | 98 | |
| 97 | 99 | static void release_uobj(struct kref *kref) |
| 98 | 100 | { |
| 99 | − kfree(container_of(kref, struct ib_uobject, ref)); | |
| 101 | + kfree_rcu(container_of(kref, struct ib_uobject, ref), rcu); | |
| 100 | 102 | } |
| 101 | 103 | |
| 102 | 104 | static void put_uobj(struct ib_uobject *uobj) |
| @@ -145,7 +147,7 @@ static struct ib_uobject *__idr_get_uobj(struct idr *idr, int id, · 42 unmodified lines | ||
| 145 | 147 | { |
| 146 | 148 | struct ib_uobject *uobj; |
| 147 | 149 | |
| 148 | − spin_lock(&ib_uverbs_idr_lock); | |
| 150 | + rcu_read_lock(); | |
| 149 | 151 | uobj = idr_find(idr, id); |
| 150 | 152 | if (uobj) { |
| 151 | 153 | if (uobj->context == context) |
| @@ -153,7 +155,7 @@ static struct ib_uobject *__idr_get_uobj(struct idr *idr, int id, · 1 unmodified line | ||
| 153 | 155 | else |
| 154 | 156 | uobj = NULL; |
| 155 | 157 | } |
| 156 | − spin_unlock(&ib_uverbs_idr_lock); | |
| 158 | + rcu_read_unlock(); | |
| 157 | 159 | |
| 158 | 160 | return uobj; |
| 159 | 161 | } |
| @@ -2446,6 +2448,7 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 2286 unmodified lines | ||
| 2446 | 2448 | int i, sg_ind; |
| 2447 | 2449 | int is_ud; |
| 2448 | 2450 | ssize_t ret = -EINVAL; |
| 2451 | + size_t next_size; | |
| 2449 | 2452 | |
| 2450 | 2453 | if (copy_from_user(&cmd, buf, sizeof cmd)) |
| 2451 | 2454 | return -EFAULT; |
| @@ -2490,7 +2493,8 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 38 unmodified lines | ||
| 2490 | 2493 | goto out_put; |
| 2491 | 2494 | } |
| 2492 | 2495 | |
| 2493 | − ud = alloc_wr(sizeof(*ud), user_wr->num_sge); | |
| 2496 | + next_size = sizeof(*ud); | |
| 2497 | + ud = alloc_wr(next_size, user_wr->num_sge); | |
| 2494 | 2498 | if (!ud) { |
| 2495 | 2499 | ret = -ENOMEM; |
| 2496 | 2500 | goto out_put; |
| @@ -2511,7 +2515,8 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 14 unmodified lines | ||
| 2511 | 2515 | user_wr->opcode == IB_WR_RDMA_READ) { |
| 2512 | 2516 | struct ib_rdma_wr *rdma; |
| 2513 | 2517 | |
| 2514 | − rdma = alloc_wr(sizeof(*rdma), user_wr->num_sge); | |
| 2518 | + next_size = sizeof(*rdma); | |
| 2519 | + rdma = alloc_wr(next_size, user_wr->num_sge); | |
| 2515 | 2520 | if (!rdma) { |
| 2516 | 2521 | ret = -ENOMEM; |
| 2517 | 2522 | goto out_put; |
| @@ -2525,7 +2530,8 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 7 unmodified lines | ||
| 2525 | 2530 | user_wr->opcode == IB_WR_ATOMIC_FETCH_AND_ADD) { |
| 2526 | 2531 | struct ib_atomic_wr *atomic; |
| 2527 | 2532 | |
| 2528 | − atomic = alloc_wr(sizeof(*atomic), user_wr->num_sge); | |
| 2533 | + next_size = sizeof(*atomic); | |
| 2534 | + atomic = alloc_wr(next_size, user_wr->num_sge); | |
| 2529 | 2535 | if (!atomic) { |
| 2530 | 2536 | ret = -ENOMEM; |
| 2531 | 2537 | goto out_put; |
| @@ -2540,7 +2546,8 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 8 unmodified lines | ||
| 2540 | 2546 | } else if (user_wr->opcode == IB_WR_SEND || |
| 2541 | 2547 | user_wr->opcode == IB_WR_SEND_WITH_IMM || |
| 2542 | 2548 | user_wr->opcode == IB_WR_SEND_WITH_INV) { |
| 2543 | − next = alloc_wr(sizeof(*next), user_wr->num_sge); | |
| 2549 | + next_size = sizeof(*next); | |
| 2550 | + next = alloc_wr(next_size, user_wr->num_sge); | |
| 2544 | 2551 | if (!next) { |
| 2545 | 2552 | ret = -ENOMEM; |
| 2546 | 2553 | goto out_put; |
| @@ -2572,7 +2579,7 @@ ssize_t ib_uverbs_post_send(struct ib_uverbs_file *file, · 25 unmodified lines | ||
| 2572 | 2579 | |
| 2573 | 2580 | if (next->num_sge) { |
| 2574 | 2581 | next->sg_list = (void *) next + |
| 2575 | − ALIGN(sizeof *next, sizeof (struct ib_sge)); | |
| 2582 | + ALIGN(next_size, sizeof(struct ib_sge)); | |
| 2576 | 2583 | if (copy_from_user(next->sg_list, |
| 2577 | 2584 | buf + sizeof cmd + |
| 2578 | 2585 | cmd.wr_count * cmd.wqe_size + |
| @@ -1516,7 +1516,7 @@ EXPORT_SYMBOL(ib_map_mr_sg); · 1515 unmodified lines | ||
| 1516 | 1516 | * @sg_nents: number of entries in sg |
| 1517 | 1517 | * @set_page: driver page assignment function pointer |
| 1518 | 1518 | * |
| 1519 | − * Core service helper for drivers to covert the largest | |
| 1519 | + * Core service helper for drivers to convert the largest | |
| 1520 | 1520 | * prefix of given sg list to a page vector. The sg list |
| 1521 | 1521 | * prefix converted is the prefix that meet the requirements |
| 1522 | 1522 | * of ib_map_mr_sg. |
| @@ -1533,7 +1533,7 @@ int ib_sg_to_pages(struct ib_mr *mr, · 10 unmodified lines | ||
| 1533 | 1533 | u64 last_end_dma_addr = 0, last_page_addr = 0; |
| 1534 | 1534 | unsigned int last_page_off = 0; |
| 1535 | 1535 | u64 page_mask = ~((u64)mr->page_size - 1); |
| 1536 | − int i; | |
| 1536 | + int i, ret; | |
| 1537 | 1537 | |
| 1538 | 1538 | mr->iova = sg_dma_address(&sgl[0]); |
| 1539 | 1539 | mr->length = 0; |
| @@ -1544,27 +1544,29 @@ int ib_sg_to_pages(struct ib_mr *mr, · 4 unmodified lines | ||
| 1544 | 1544 | u64 end_dma_addr = dma_addr + dma_len; |
| 1545 | 1545 | u64 page_addr = dma_addr & page_mask; |
| 1546 | 1546 | |
| 1547 | − if (i && page_addr != dma_addr) { | |
| 1548 | − if (last_end_dma_addr != dma_addr) { | |
| 1549 | − /* gap */ | |
| 1550 | − goto done; | |
| 1551 | − | |
| 1552 | − } else if (last_page_off + dma_len <= mr->page_size) { | |
| 1553 | − /* chunk this fragment with the last */ | |
| 1554 | − mr->length += dma_len; | |
| 1555 | − last_end_dma_addr += dma_len; | |
| 1556 | − last_page_off += dma_len; | |
| 1557 | − continue; | |
| 1558 | − } else { | |
| 1559 | − /* map starting from the next page */ | |
| 1560 | − page_addr = last_page_addr + mr->page_size; | |
| 1561 | − dma_len -= mr->page_size - last_page_off; | |
| 1562 | − } | |
| 1547 | + /* | |
| 1548 | + * For the second and later elements, check whether either the | |
| 1549 | + * end of element i-1 or the start of element i is not aligned | |
| 1550 | + * on a page boundary. | |
| 1551 | + */ | |
| 1552 | + if (i && (last_page_off != 0 || page_addr != dma_addr)) { | |
| 1553 | + /* Stop mapping if there is a gap. */ | |
| 1554 | + if (last_end_dma_addr != dma_addr) | |
| 1555 | + break; | |
| 1556 | + | |
| 1557 | + /* | |
| 1558 | + * Coalesce this element with the last. If it is small | |
| 1559 | + * enough just update mr->length. Otherwise start | |
| 1560 | + * mapping from the next page. | |
| 1561 | + */ | |
| 1562 | + goto next_page; | |
| 1563 | 1563 | } |
| 1564 | 1564 | |
| 1565 | 1565 | do { |
| 1566 | − if (unlikely(set_page(mr, page_addr))) | |
| 1567 | − goto done; | |
| 1566 | + ret = set_page(mr, page_addr); | |
| 1567 | + if (unlikely(ret < 0)) | |
| 1568 | + return i ? : ret; | |
| 1569 | +next_page: | |
| 1568 | 1570 | page_addr += mr->page_size; |
| 1569 | 1571 | } while (page_addr < end_dma_addr); |
| 1570 | 1572 | |
| @@ -1574,7 +1576,6 @@ int ib_sg_to_pages(struct ib_mr *mr, · 3 unmodified lines | ||
| 1574 | 1576 | last_page_off = end_dma_addr & ~page_mask; |
| 1575 | 1577 | } |
| 1576 | 1578 | |
| 1577 | −done: | |
| 1578 | 1579 | return i; |
| 1579 | 1580 | } |
| 1580 | 1581 | EXPORT_SYMBOL(ib_sg_to_pages); |
| @@ -456,7 +456,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, · 455 unmodified lines | ||
| 456 | 456 | props->max_qp_wr = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE; |
| 457 | 457 | props->max_sge = min(dev->dev->caps.max_sq_sg, |
| 458 | 458 | dev->dev->caps.max_rq_sg); |
| 459 | − props->max_sge_rd = props->max_sge; | |
| 459 | + props->max_sge_rd = MLX4_MAX_SGE_RD; | |
| 460 | 460 | props->max_cq = dev->dev->quotas.cq; |
| 461 | 461 | props->max_cqe = dev->dev->caps.max_cqes; |
| 462 | 462 | props->max_mr = dev->dev->quotas.mpt; |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <linux/log2.h> |
| 35 | 35 | #include <linux/slab.h> |
| 36 | 36 | #include <linux/netdevice.h> |
| 37 | +#include <linux/vmalloc.h> | |
| 37 | 38 | |
| 38 | 39 | #include <rdma/ib_cache.h> |
| 39 | 40 | #include <rdma/ib_pack.h> |
| @@ -795,8 +796,14 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, · 755 unmodified lines | ||
| 795 | 796 | if (err) |
| 796 | 797 | goto err_mtt; |
| 797 | 798 | |
| 798 | − qp->sq.wrid = kmalloc(qp->sq.wqe_cnt * sizeof (u64), gfp); | |
| 799 | − qp->rq.wrid = kmalloc(qp->rq.wqe_cnt * sizeof (u64), gfp); | |
| 799 | + qp->sq.wrid = kmalloc(qp->sq.wqe_cnt * sizeof(u64), gfp); | |
| 800 | + if (!qp->sq.wrid) | |
| 801 | + qp->sq.wrid = __vmalloc(qp->sq.wqe_cnt * sizeof(u64), | |
| 802 | + gfp, PAGE_KERNEL); | |
| 803 | + qp->rq.wrid = kmalloc(qp->rq.wqe_cnt * sizeof(u64), gfp); | |
| 804 | + if (!qp->rq.wrid) | |
| 805 | + qp->rq.wrid = __vmalloc(qp->rq.wqe_cnt * sizeof(u64), | |
| 806 | + gfp, PAGE_KERNEL); | |
| 800 | 807 | if (!qp->sq.wrid || !qp->rq.wrid) { |
| 801 | 808 | err = -ENOMEM; |
| 802 | 809 | goto err_wrid; |
| @@ -886,8 +893,8 @@ err_wrid: · 83 unmodified lines | ||
| 886 | 893 | if (qp_has_rq(init_attr)) |
| 887 | 894 | mlx4_ib_db_unmap_user(to_mucontext(pd->uobject->context), &qp->db); |
| 888 | 895 | } else { |
| 889 | − kfree(qp->sq.wrid); | |
| 890 | − kfree(qp->rq.wrid); | |
| 896 | + kvfree(qp->sq.wrid); | |
| 897 | + kvfree(qp->rq.wrid); | |
| 891 | 898 | } |
| 892 | 899 | |
| 893 | 900 | err_mtt: |
| @@ -1062,8 +1069,8 @@ static void destroy_qp_common(struct mlx4_ib_dev *dev, struct mlx4_ib_qp *qp, · 168 unmodified lines | ||
| 1062 | 1069 | &qp->db); |
| 1063 | 1070 | ib_umem_release(qp->umem); |
| 1064 | 1071 | } else { |
| 1065 | − kfree(qp->sq.wrid); | |
| 1066 | − kfree(qp->rq.wrid); | |
| 1072 | + kvfree(qp->sq.wrid); | |
| 1073 | + kvfree(qp->rq.wrid); | |
| 1067 | 1074 | if (qp->mlx4_ib_qp_type & (MLX4_IB_QPT_PROXY_SMI_OWNER | |
| 1068 | 1075 | MLX4_IB_QPT_PROXY_SMI | MLX4_IB_QPT_PROXY_GSI)) |
| 1069 | 1076 | free_proxy_bufs(&dev->ib_dev, qp); |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <linux/mlx4/qp.h> |
| 35 | 35 | #include <linux/mlx4/srq.h> |
| 36 | 36 | #include <linux/slab.h> |
| 37 | +#include <linux/vmalloc.h> | |
| 37 | 38 | |
| 38 | 39 | #include "mlx4_ib.h" |
| 39 | 40 | #include "user.h" |
| @@ -172,8 +173,12 @@ struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, · 132 unmodified lines | ||
| 172 | 173 | |
| 173 | 174 | srq->wrid = kmalloc(srq->msrq.max * sizeof (u64), GFP_KERNEL); |
| 174 | 175 | if (!srq->wrid) { |
| 175 | − err = -ENOMEM; | |
| 176 | − goto err_mtt; | |
| 176 | + srq->wrid = __vmalloc(srq->msrq.max * sizeof(u64), | |
| 177 | + GFP_KERNEL, PAGE_KERNEL); | |
| 178 | + if (!srq->wrid) { | |
| 179 | + err = -ENOMEM; | |
| 180 | + goto err_mtt; | |
| 181 | + } | |
| 177 | 182 | } |
| 178 | 183 | } |
| 179 | 184 | |
| @@ -204,7 +209,7 @@ err_wrid: · 24 unmodified lines | ||
| 204 | 209 | if (pd->uobject) |
| 205 | 210 | mlx4_ib_db_unmap_user(to_mucontext(pd->uobject->context), &srq->db); |
| 206 | 211 | else |
| 207 | − kfree(srq->wrid); | |
| 212 | + kvfree(srq->wrid); | |
| 208 | 213 | |
| 209 | 214 | err_mtt: |
| 210 | 215 | mlx4_mtt_cleanup(dev->dev, &srq->mtt); |
| @@ -381,7 +381,19 @@ static void __cache_work_func(struct mlx5_cache_ent *ent) · 380 unmodified lines | ||
| 381 | 381 | } |
| 382 | 382 | } |
| 383 | 383 | } else if (ent->cur > 2 * ent->limit) { |
| 384 | − if (!someone_adding(cache) && | |
| 384 | + /* | |
| 385 | + * The remove_keys() logic is performed as garbage collection | |
| 386 | + * task. Such task is intended to be run when no other active | |
| 387 | + * processes are running. | |
| 388 | + * | |
| 389 | + * The need_resched() will return TRUE if there are user tasks | |
| 390 | + * to be activated in near future. | |
| 391 | + * | |
| 392 | + * In such case, we don't execute remove_keys() and postpone | |
| 393 | + * the garbage collection work to try to run in next cycle, | |
| 394 | + * in order to free CPU resources to other tasks. | |
| 395 | + */ | |
| 396 | + if (!need_resched() && !someone_adding(cache) && | |
| 385 | 397 | time_after(jiffies, cache->last_add + 300 * HZ)) { |
| 386 | 398 | remove_keys(dev, i, 1); |
| 387 | 399 | if (ent->cur > ent->limit) |
| @@ -292,7 +292,7 @@ int qib_refresh_qsfp_cache(struct qib_pportdata *ppd, struct qib_qsfp_cache *cp) · 291 unmodified lines | ||
| 292 | 292 | qib_dev_porterr(ppd->dd, ppd->port, |
| 293 | 293 | "QSFP byte0 is 0x%02X, S/B 0x0C/D\n", peek[0]); |
| 294 | 294 | |
| 295 | − if ((peek[2] & 2) == 0) { | |
| 295 | + if ((peek[2] & 4) == 0) { | |
| 296 | 296 | /* |
| 297 | 297 | * If cable is paged, rather than "flat memory", we need to |
| 298 | 298 | * set the page to zero, Even if it already appears to be zero. |
| @@ -538,7 +538,7 @@ int qib_qsfp_dump(struct qib_pportdata *ppd, char *buf, int len) · 239 unmodified lines | ||
| 538 | 538 | sofar += scnprintf(buf + sofar, len - sofar, "Date:%.*s\n", |
| 539 | 539 | QSFP_DATE_LEN, cd.date); |
| 540 | 540 | sofar += scnprintf(buf + sofar, len - sofar, "Lot:%.*s\n", |
| 541 | − QSFP_LOT_LEN, cd.date); | |
| 541 | + QSFP_LOT_LEN, cd.lot); | |
| 542 | 542 | |
| 543 | 543 | while (bidx < QSFP_DEFAULT_HDR_CNT) { |
| 544 | 544 | int iidx; |
| @@ -329,9 +329,9 @@ struct qib_sge { · 328 unmodified lines | ||
| 329 | 329 | struct qib_mr { |
| 330 | 330 | struct ib_mr ibmr; |
| 331 | 331 | struct ib_umem *umem; |
| 332 | − struct qib_mregion mr; /* must be last */ | |
| 333 | 332 | u64 *pages; |
| 334 | 333 | u32 npages; |
| 334 | + struct qib_mregion mr; /* must be last */ | |
| 335 | 335 | }; |
| 336 | 336 | |
| 337 | 337 | /* |
| @@ -1293,7 +1293,7 @@ u8 iser_check_task_pi_status(struct iscsi_iser_task *iser_task, · 1292 unmodified lines | ||
| 1293 | 1293 | if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) { |
| 1294 | 1294 | sector_t sector_off = mr_status.sig_err.sig_err_offset; |
| 1295 | 1295 | |
| 1296 | − do_div(sector_off, sector_size + 8); | |
| 1296 | + sector_div(sector_off, sector_size + 8); | |
| 1297 | 1297 | *sector = scsi_get_lba(iser_task->sc) + sector_off; |
| 1298 | 1298 | |
| 1299 | 1299 | pr_err("PI error found type %d at sector %llx " |
| @@ -157,16 +157,9 @@ isert_create_qp(struct isert_conn *isert_conn, · 156 unmodified lines | ||
| 157 | 157 | attr.recv_cq = comp->cq; |
| 158 | 158 | attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS; |
| 159 | 159 | attr.cap.max_recv_wr = ISERT_QP_MAX_RECV_DTOS + 1; |
| 160 | − /* | |
| 161 | − * FIXME: Use devattr.max_sge - 2 for max_send_sge as | |
| 162 | − * work-around for RDMA_READs with ConnectX-2. | |
| 163 | − * | |
| 164 | − * Also, still make sure to have at least two SGEs for | |
| 165 | − * outgoing control PDU responses. | |
| 166 | − */ | |
| 167 | − attr.cap.max_send_sge = max(2, device->dev_attr.max_sge - 2); | |
| 168 | − isert_conn->max_sge = attr.cap.max_send_sge; | |
| 169 | − | |
| 160 | + attr.cap.max_send_sge = device->dev_attr.max_sge; | |
| 161 | + isert_conn->max_sge = min(device->dev_attr.max_sge, | |
| 162 | + device->dev_attr.max_sge_rd); | |
| 170 | 163 | attr.cap.max_recv_sge = 1; |
| 171 | 164 | attr.sq_sig_type = IB_SIGNAL_REQ_WR; |
| 172 | 165 | attr.qp_type = IB_QPT_RC; |
| @@ -488,7 +488,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) · 487 unmodified lines | ||
| 488 | 488 | struct ib_qp *qp; |
| 489 | 489 | struct ib_fmr_pool *fmr_pool = NULL; |
| 490 | 490 | struct srp_fr_pool *fr_pool = NULL; |
| 491 | − const int m = 1 + dev->use_fast_reg; | |
| 491 | + const int m = dev->use_fast_reg ? 3 : 1; | |
| 492 | 492 | struct ib_cq_init_attr cq_attr = {}; |
| 493 | 493 | int ret; |
| 494 | 494 | |
| @@ -994,16 +994,16 @@ static int srp_connect_ch(struct srp_rdma_ch *ch, bool multich) · 499 unmodified lines | ||
| 994 | 994 | |
| 995 | 995 | ret = srp_lookup_path(ch); |
| 996 | 996 | if (ret) |
| 997 | − return ret; | |
| 997 | + goto out; | |
| 998 | 998 | |
| 999 | 999 | while (1) { |
| 1000 | 1000 | init_completion(&ch->done); |
| 1001 | 1001 | ret = srp_send_req(ch, multich); |
| 1002 | 1002 | if (ret) |
| 1003 | − return ret; | |
| 1003 | + goto out; | |
| 1004 | 1004 | ret = wait_for_completion_interruptible(&ch->done); |
| 1005 | 1005 | if (ret < 0) |
| 1006 | − return ret; | |
| 1006 | + goto out; | |
| 1007 | 1007 | |
| 1008 | 1008 | /* |
| 1009 | 1009 | * The CM event handling code will set status to |
| @@ -1011,15 +1011,16 @@ static int srp_connect_ch(struct srp_rdma_ch *ch, bool multich) · 1 unmodified line | ||
| 1011 | 1011 | * back, or SRP_DLID_REDIRECT if we get a lid/qp |
| 1012 | 1012 | * redirect REJ back. |
| 1013 | 1013 | */ |
| 1014 | − switch (ch->status) { | |
| 1014 | + ret = ch->status; | |
| 1015 | + switch (ret) { | |
| 1015 | 1016 | case 0: |
| 1016 | 1017 | ch->connected = true; |
| 1017 | − return 0; | |
| 1018 | + goto out; | |
| 1018 | 1019 | |
| 1019 | 1020 | case SRP_PORT_REDIRECT: |
| 1020 | 1021 | ret = srp_lookup_path(ch); |
| 1021 | 1022 | if (ret) |
| 1022 | − return ret; | |
| 1023 | + goto out; | |
| 1023 | 1024 | break; |
| 1024 | 1025 | |
| 1025 | 1026 | case SRP_DLID_REDIRECT: |
| @@ -1028,13 +1029,16 @@ static int srp_connect_ch(struct srp_rdma_ch *ch, bool multich) · 2 unmodified lines | ||
| 1028 | 1029 | case SRP_STALE_CONN: |
| 1029 | 1030 | shost_printk(KERN_ERR, target->scsi_host, PFX |
| 1030 | 1031 | "giving up on stale connection\n"); |
| 1031 | − ch->status = -ECONNRESET; | |
| 1032 | − return ch->status; | |
| 1032 | + ret = -ECONNRESET; | |
| 1033 | + goto out; | |
| 1033 | 1034 | |
| 1034 | 1035 | default: |
| 1035 | − return ch->status; | |
| 1036 | + goto out; | |
| 1036 | 1037 | } |
| 1037 | 1038 | } |
| 1039 | + | |
| 1040 | +out: | |
| 1041 | + return ret <= 0 ? ret : -ENODEV; | |
| 1038 | 1042 | } |
| 1039 | 1043 | |
| 1040 | 1044 | static int srp_inv_rkey(struct srp_rdma_ch *ch, u32 rkey) |
| @@ -1309,7 +1313,7 @@ reset_state: · 268 unmodified lines | ||
| 1309 | 1313 | } |
| 1310 | 1314 | |
| 1311 | 1315 | static int srp_map_finish_fr(struct srp_map_state *state, |
| 1312 | − struct srp_rdma_ch *ch) | |
| 1316 | + struct srp_rdma_ch *ch, int sg_nents) | |
| 1313 | 1317 | { |
| 1314 | 1318 | struct srp_target_port *target = ch->target; |
| 1315 | 1319 | struct srp_device *dev = target->srp_host->srp_dev; |
| @@ -1324,10 +1328,10 @@ static int srp_map_finish_fr(struct srp_map_state *state, · 8 unmodified lines | ||
| 1324 | 1328 | |
| 1325 | 1329 | WARN_ON_ONCE(!dev->use_fast_reg); |
| 1326 | 1330 | |
| 1327 | − if (state->sg_nents == 0) | |
| 1331 | + if (sg_nents == 0) | |
| 1328 | 1332 | return 0; |
| 1329 | 1333 | |
| 1330 | − if (state->sg_nents == 1 && target->global_mr) { | |
| 1334 | + if (sg_nents == 1 && target->global_mr) { | |
| 1331 | 1335 | srp_map_desc(state, sg_dma_address(state->sg), |
| 1332 | 1336 | sg_dma_len(state->sg), |
| 1333 | 1337 | target->global_mr->rkey); |
| @@ -1341,8 +1345,7 @@ static int srp_map_finish_fr(struct srp_map_state *state, · 7 unmodified lines | ||
| 1341 | 1345 | rkey = ib_inc_rkey(desc->mr->rkey); |
| 1342 | 1346 | ib_update_fast_reg_key(desc->mr, rkey); |
| 1343 | 1347 | |
| 1344 | − n = ib_map_mr_sg(desc->mr, state->sg, state->sg_nents, | |
| 1345 | − dev->mr_page_size); | |
| 1348 | + n = ib_map_mr_sg(desc->mr, state->sg, sg_nents, dev->mr_page_size); | |
| 1346 | 1349 | if (unlikely(n < 0)) |
| 1347 | 1350 | return n; |
| 1348 | 1351 | |
| @@ -1448,16 +1451,15 @@ static int srp_map_sg_fr(struct srp_map_state *state, struct srp_rdma_ch *ch, · 99 unmodified lines | ||
| 1448 | 1451 | state->fr.next = req->fr_list; |
| 1449 | 1452 | state->fr.end = req->fr_list + ch->target->cmd_sg_cnt; |
| 1450 | 1453 | state->sg = scat; |
| 1451 | − state->sg_nents = scsi_sg_count(req->scmnd); | |
| 1452 | 1454 | |
| 1453 | − while (state->sg_nents) { | |
| 1455 | + while (count) { | |
| 1454 | 1456 | int i, n; |
| 1455 | 1457 | |
| 1456 | − n = srp_map_finish_fr(state, ch); | |
| 1458 | + n = srp_map_finish_fr(state, ch, count); | |
| 1457 | 1459 | if (unlikely(n < 0)) |
| 1458 | 1460 | return n; |
| 1459 | 1461 | |
| 1460 | − state->sg_nents -= n; | |
| 1462 | + count -= n; | |
| 1461 | 1463 | for (i = 0; i < n; i++) |
| 1462 | 1464 | state->sg = sg_next(state->sg); |
| 1463 | 1465 | } |
| @@ -1517,10 +1519,12 @@ static int srp_map_idb(struct srp_rdma_ch *ch, struct srp_request *req, · 53 unmodified lines | ||
| 1517 | 1519 | |
| 1518 | 1520 | if (dev->use_fast_reg) { |
| 1519 | 1521 | state.sg = idb_sg; |
| 1520 | − state.sg_nents = 1; | |
| 1521 | 1522 | sg_set_buf(idb_sg, req->indirect_desc, idb_len); |
| 1522 | 1523 | idb_sg->dma_address = req->indirect_dma_addr; /* hack! */ |
| 1523 | − ret = srp_map_finish_fr(&state, ch); | |
| 1524 | +#ifdef CONFIG_NEED_SG_DMA_LENGTH | |
| 1525 | + idb_sg->dma_length = idb_sg->length; /* hack^2 */ | |
| 1526 | +#endif | |
| 1527 | + ret = srp_map_finish_fr(&state, ch, 1); | |
| 1524 | 1528 | if (ret < 0) |
| 1525 | 1529 | return ret; |
| 1526 | 1530 | } else if (dev->use_fmr) { |
| @@ -1655,7 +1659,7 @@ static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_rdma_ch *ch, · 128 unmodified lines | ||
| 1655 | 1659 | return ret; |
| 1656 | 1660 | req->nmdesc++; |
| 1657 | 1661 | } else { |
| 1658 | − idb_rkey = target->global_mr->rkey; | |
| 1662 | + idb_rkey = cpu_to_be32(target->global_mr->rkey); | |
| 1659 | 1663 | } |
| 1660 | 1664 | |
| 1661 | 1665 | indirect_hdr->table_desc.va = cpu_to_be64(req->indirect_dma_addr); |
| @@ -300,10 +300,7 @@ struct srp_map_state { · 299 unmodified lines | ||
| 300 | 300 | dma_addr_t base_dma_addr; |
| 301 | 301 | u32 dma_len; |
| 302 | 302 | u32 total_len; |
| 303 | − union { | |
| 304 | − unsigned int npages; | |
| 305 | − int sg_nents; | |
| 306 | − }; | |
| 303 | + unsigned int npages; | |
| 307 | 304 | unsigned int nmdesc; |
| 308 | 305 | unsigned int ndesc; |
| 309 | 306 | }; |
| @@ -210,7 +210,12 @@ int __init fpga_irq_of_init(struct device_node *node, · 209 unmodified lines | ||
| 210 | 210 | parent_irq = -1; |
| 211 | 211 | } |
| 212 | 212 | |
| 213 | +#ifdef CONFIG_ARCH_VERSATILE | |
| 214 | + fpga_irq_init(base, node->name, IRQ_SIC_START, parent_irq, valid_mask, | |
| 215 | + node); | |
| 216 | +#else | |
| 213 | 217 | fpga_irq_init(base, node->name, 0, parent_irq, valid_mask, node); |
| 218 | +#endif | |
| 214 | 219 | |
| 215 | 220 | writel(clear_mask, base + IRQ_ENABLE_CLEAR); |
| 216 | 221 | writel(clear_mask, base + FIQ_ENABLE_CLEAR); |
| @@ -1896,7 +1896,7 @@ static void EChannel_proc_rcv(struct hisax_d_if *d_if) · 1895 unmodified lines | ||
| 1896 | 1896 | ptr--; |
| 1897 | 1897 | *ptr++ = '\n'; |
| 1898 | 1898 | *ptr = 0; |
| 1899 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 1899 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 1900 | 1900 | } else |
| 1901 | 1901 | HiSax_putstatus(cs, "LogEcho: ", |
| 1902 | 1902 | "warning Frame too big (%d)", |
| @@ -901,7 +901,7 @@ Begin: · 900 unmodified lines | ||
| 901 | 901 | ptr--; |
| 902 | 902 | *ptr++ = '\n'; |
| 903 | 903 | *ptr = 0; |
| 904 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 904 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 905 | 905 | } else |
| 906 | 906 | HiSax_putstatus(cs, "LogEcho: ", "warning Frame too big (%d)", total - 3); |
| 907 | 907 | } |
| @@ -674,7 +674,7 @@ receive_emsg(struct IsdnCardState *cs) · 673 unmodified lines | ||
| 674 | 674 | ptr--; |
| 675 | 675 | *ptr++ = '\n'; |
| 676 | 676 | *ptr = 0; |
| 677 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 677 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 678 | 678 | } else |
| 679 | 679 | HiSax_putstatus(cs, "LogEcho: ", "warning Frame too big (%d)", skb->len); |
| 680 | 680 | } |
| @@ -1179,7 +1179,7 @@ LogFrame(struct IsdnCardState *cs, u_char *buf, int size) · 1178 unmodified lines | ||
| 1179 | 1179 | dp--; |
| 1180 | 1180 | *dp++ = '\n'; |
| 1181 | 1181 | *dp = 0; |
| 1182 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 1182 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 1183 | 1183 | } else |
| 1184 | 1184 | HiSax_putstatus(cs, "LogFrame: ", "warning Frame too big (%d)", size); |
| 1185 | 1185 | } |
| @@ -1246,7 +1246,7 @@ dlogframe(struct IsdnCardState *cs, struct sk_buff *skb, int dir) · 60 unmodified lines | ||
| 1246 | 1246 | } |
| 1247 | 1247 | if (finish) { |
| 1248 | 1248 | *dp = 0; |
| 1249 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 1249 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 1250 | 1250 | return; |
| 1251 | 1251 | } |
| 1252 | 1252 | if ((0xfe & buf[0]) == PROTO_DIS_N0) { /* 1TR6 */ |
| @@ -1509,5 +1509,5 @@ dlogframe(struct IsdnCardState *cs, struct sk_buff *skb, int dir) · 256 unmodified lines | ||
| 1509 | 1509 | dp += sprintf(dp, "Unknown protocol %x!", buf[0]); |
| 1510 | 1510 | } |
| 1511 | 1511 | *dp = 0; |
| 1512 | − HiSax_putstatus(cs, NULL, "%s", cs->dlog); | |
| 1512 | + HiSax_putstatus(cs, NULL, cs->dlog); | |
| 1513 | 1513 | } |
| @@ -18,6 +18,7 @@ if NVM · 17 unmodified lines | ||
| 18 | 18 | |
| 19 | 19 | config NVM_DEBUG |
| 20 | 20 | bool "Open-Channel SSD debugging support" |
| 21 | + default n | |
| 21 | 22 | ---help--- |
| 22 | 23 | Exposes a debug management interface to create/remove targets at: |
| 23 | 24 | |
| @@ -74,7 +74,7 @@ EXPORT_SYMBOL(nvm_unregister_target); · 73 unmodified lines | ||
| 74 | 74 | void *nvm_dev_dma_alloc(struct nvm_dev *dev, gfp_t mem_flags, |
| 75 | 75 | dma_addr_t *dma_handler) |
| 76 | 76 | { |
| 77 | − return dev->ops->dev_dma_alloc(dev->q, dev->ppalist_pool, mem_flags, | |
| 77 | + return dev->ops->dev_dma_alloc(dev, dev->ppalist_pool, mem_flags, | |
| 78 | 78 | dma_handler); |
| 79 | 79 | } |
| 80 | 80 | EXPORT_SYMBOL(nvm_dev_dma_alloc); |
| @@ -97,15 +97,47 @@ static struct nvmm_type *nvm_find_mgr_type(const char *name) · 16 unmodified lines | ||
| 97 | 97 | return NULL; |
| 98 | 98 | } |
| 99 | 99 | |
| 100 | +struct nvmm_type *nvm_init_mgr(struct nvm_dev *dev) | |
| 101 | +{ | |
| 102 | + struct nvmm_type *mt; | |
| 103 | + int ret; | |
| 104 | + | |
| 105 | + lockdep_assert_held(&nvm_lock); | |
| 106 | + | |
| 107 | + list_for_each_entry(mt, &nvm_mgrs, list) { | |
| 108 | + ret = mt->register_mgr(dev); | |
| 109 | + if (ret < 0) { | |
| 110 | + pr_err("nvm: media mgr failed to init (%d) on dev %s\n", | |
| 111 | + ret, dev->name); | |
| 112 | + return NULL; /* initialization failed */ | |
| 113 | + } else if (ret > 0) | |
| 114 | + return mt; | |
| 115 | + } | |
| 116 | + | |
| 117 | + return NULL; | |
| 118 | +} | |
| 119 | + | |
| 100 | 120 | int nvm_register_mgr(struct nvmm_type *mt) |
| 101 | 121 | { |
| 122 | + struct nvm_dev *dev; | |
| 102 | 123 | int ret = 0; |
| 103 | 124 | |
| 104 | 125 | down_write(&nvm_lock); |
| 105 | − if (nvm_find_mgr_type(mt->name)) | |
| 126 | + if (nvm_find_mgr_type(mt->name)) { | |
| 106 | 127 | ret = -EEXIST; |
| 107 | − else | |
| 128 | + goto finish; | |
| 129 | + } else { | |
| 108 | 130 | list_add(&mt->list, &nvm_mgrs); |
| 131 | + } | |
| 132 | + | |
| 133 | + /* try to register media mgr if any device have none configured */ | |
| 134 | + list_for_each_entry(dev, &nvm_devices, devices) { | |
| 135 | + if (dev->mt) | |
| 136 | + continue; | |
| 137 | + | |
| 138 | + dev->mt = nvm_init_mgr(dev); | |
| 139 | + } | |
| 140 | +finish: | |
| 109 | 141 | up_write(&nvm_lock); |
| 110 | 142 | |
| 111 | 143 | return ret; |
| @@ -160,11 +192,6 @@ int nvm_erase_blk(struct nvm_dev *dev, struct nvm_block *blk) · 48 unmodified lines | ||
| 160 | 192 | } |
| 161 | 193 | EXPORT_SYMBOL(nvm_erase_blk); |
| 162 | 194 | |
| 163 | −static void nvm_core_free(struct nvm_dev *dev) | |
| 164 | −{ | |
| 165 | − kfree(dev); | |
| 166 | −} | |
| 167 | − | |
| 168 | 195 | static int nvm_core_init(struct nvm_dev *dev) |
| 169 | 196 | { |
| 170 | 197 | struct nvm_id *id = &dev->identity; |
| @@ -179,12 +206,21 @@ static int nvm_core_init(struct nvm_dev *dev) · 8 unmodified lines | ||
| 179 | 206 | dev->sec_size = grp->csecs; |
| 180 | 207 | dev->oob_size = grp->sos; |
| 181 | 208 | dev->sec_per_pg = grp->fpg_sz / grp->csecs; |
| 182 | − dev->addr_mode = id->ppat; | |
| 183 | − dev->addr_format = id->ppaf; | |
| 209 | + memcpy(&dev->ppaf, &id->ppaf, sizeof(struct nvm_addr_format)); | |
| 184 | 210 | |
| 185 | 211 | dev->plane_mode = NVM_PLANE_SINGLE; |
| 186 | 212 | dev->max_rq_size = dev->ops->max_phys_sect * dev->sec_size; |
| 187 | 213 | |
| 214 | + if (grp->mtype != 0) { | |
| 215 | + pr_err("nvm: memory type not supported\n"); | |
| 216 | + return -EINVAL; | |
| 217 | + } | |
| 218 | + | |
| 219 | + if (grp->fmtype != 0 && grp->fmtype != 1) { | |
| 220 | + pr_err("nvm: flash type not supported\n"); | |
| 221 | + return -EINVAL; | |
| 222 | + } | |
| 223 | + | |
| 188 | 224 | if (grp->mpos & 0x020202) |
| 189 | 225 | dev->plane_mode = NVM_PLANE_DOUBLE; |
| 190 | 226 | if (grp->mpos & 0x040404) |
| @@ -213,21 +249,17 @@ static void nvm_free(struct nvm_dev *dev) · 22 unmodified lines | ||
| 213 | 249 | |
| 214 | 250 | if (dev->mt) |
| 215 | 251 | dev->mt->unregister_mgr(dev); |
| 216 | − | |
| 217 | − nvm_core_free(dev); | |
| 218 | 252 | } |
| 219 | 253 | |
| 220 | 254 | static int nvm_init(struct nvm_dev *dev) |
| 221 | 255 | { |
| 222 | − struct nvmm_type *mt; | |
| 223 | − int ret = 0; | |
| 256 | + int ret = -EINVAL; | |
| 224 | 257 | |
| 225 | 258 | if (!dev->q || !dev->ops) |
| 226 | − return -EINVAL; | |
| 259 | + return ret; | |
| 227 | 260 | |
| 228 | − if (dev->ops->identity(dev->q, &dev->identity)) { | |
| 261 | + if (dev->ops->identity(dev, &dev->identity)) { | |
| 229 | 262 | pr_err("nvm: device could not be identified\n"); |
| 230 | − ret = -EINVAL; | |
| 231 | 263 | goto err; |
| 232 | 264 | } |
| 233 | 265 | |
| @@ -251,29 +283,12 @@ static int nvm_init(struct nvm_dev *dev) · 17 unmodified lines | ||
| 251 | 283 | goto err; |
| 252 | 284 | } |
| 253 | 285 | |
| 254 | − /* register with device with a supported manager */ | |
| 255 | − list_for_each_entry(mt, &nvm_mgrs, list) { | |
| 256 | − ret = mt->register_mgr(dev); | |
| 257 | − if (ret < 0) | |
| 258 | − goto err; /* initialization failed */ | |
| 259 | − if (ret > 0) { | |
| 260 | − dev->mt = mt; | |
| 261 | − break; /* successfully initialized */ | |
| 262 | − } | |
| 263 | − } | |
| 264 | − | |
| 265 | − if (!ret) { | |
| 266 | − pr_info("nvm: no compatible manager found.\n"); | |
| 267 | − return 0; | |
| 268 | − } | |
| 269 | − | |
| 270 | 286 | pr_info("nvm: registered %s [%u/%u/%u/%u/%u/%u]\n", |
| 271 | 287 | dev->name, dev->sec_per_pg, dev->nr_planes, |
| 272 | 288 | dev->pgs_per_blk, dev->blks_per_lun, dev->nr_luns, |
| 273 | 289 | dev->nr_chnls); |
| 274 | 290 | return 0; |
| 275 | 291 | err: |
| 276 | − nvm_free(dev); | |
| 277 | 292 | pr_err("nvm: failed to initialize nvm\n"); |
| 278 | 293 | return ret; |
| 279 | 294 | } |
| @@ -308,22 +323,27 @@ int nvm_register(struct request_queue *q, char *disk_name, · 28 unmodified lines | ||
| 308 | 323 | if (ret) |
| 309 | 324 | goto err_init; |
| 310 | 325 | |
| 311 | − down_write(&nvm_lock); | |
| 312 | − list_add(&dev->devices, &nvm_devices); | |
| 313 | − up_write(&nvm_lock); | |
| 326 | + if (dev->ops->max_phys_sect > 256) { | |
| 327 | + pr_info("nvm: max sectors supported is 256.\n"); | |
| 328 | + ret = -EINVAL; | |
| 329 | + goto err_init; | |
| 330 | + } | |
| 314 | 331 | |
| 315 | 332 | if (dev->ops->max_phys_sect > 1) { |
| 316 | − dev->ppalist_pool = dev->ops->create_dma_pool(dev->q, | |
| 317 | − "ppalist"); | |
| 333 | + dev->ppalist_pool = dev->ops->create_dma_pool(dev, "ppalist"); | |
| 318 | 334 | if (!dev->ppalist_pool) { |
| 319 | 335 | pr_err("nvm: could not create ppa pool\n"); |
| 320 | − return -ENOMEM; | |
| 336 | + ret = -ENOMEM; | |
| 337 | + goto err_init; | |
| 321 | 338 | } |
| 322 | − } else if (dev->ops->max_phys_sect > 256) { | |
| 323 | − pr_info("nvm: max sectors supported is 256.\n"); | |
| 324 | − return -EINVAL; | |
| 325 | 339 | } |
| 326 | 340 | |
| 341 | + /* register device with a supported media manager */ | |
| 342 | + down_write(&nvm_lock); | |
| 343 | + dev->mt = nvm_init_mgr(dev); | |
| 344 | + list_add(&dev->devices, &nvm_devices); | |
| 345 | + up_write(&nvm_lock); | |
| 346 | + | |
| 327 | 347 | return 0; |
| 328 | 348 | err_init: |
| 329 | 349 | kfree(dev); |
| @@ -333,19 +353,22 @@ EXPORT_SYMBOL(nvm_register); · 3 unmodified lines | ||
| 333 | 353 | |
| 334 | 354 | void nvm_unregister(char *disk_name) |
| 335 | 355 | { |
| 336 | − struct nvm_dev *dev = nvm_find_nvm_dev(disk_name); | |
| 356 | + struct nvm_dev *dev; | |
| 337 | 357 | |
| 358 | + down_write(&nvm_lock); | |
| 359 | + dev = nvm_find_nvm_dev(disk_name); | |
| 338 | 360 | if (!dev) { |
| 339 | 361 | pr_err("nvm: could not find device %s to unregister\n", |
| 340 | 362 | disk_name); |
| 363 | + up_write(&nvm_lock); | |
| 341 | 364 | return; |
| 342 | 365 | } |
| 343 | 366 | |
| 344 | − nvm_exit(dev); | |
| 345 | − | |
| 346 | − down_write(&nvm_lock); | |
| 347 | 367 | list_del(&dev->devices); |
| 348 | 368 | up_write(&nvm_lock); |
| 369 | + | |
| 370 | + nvm_exit(dev); | |
| 371 | + kfree(dev); | |
| 349 | 372 | } |
| 350 | 373 | EXPORT_SYMBOL(nvm_unregister); |
| 351 | 374 | |
| @@ -358,38 +381,24 @@ static int nvm_create_target(struct nvm_dev *dev, · 6 unmodified lines | ||
| 358 | 381 | { |
| 359 | 382 | struct nvm_ioctl_create_simple *s = &create->conf.s; |
| 360 | 383 | struct request_queue *tqueue; |
| 361 | − struct nvmm_type *mt; | |
| 362 | 384 | struct gendisk *tdisk; |
| 363 | 385 | struct nvm_tgt_type *tt; |
| 364 | 386 | struct nvm_target *t; |
| 365 | 387 | void *targetdata; |
| 366 | − int ret = 0; | |
| 367 | 388 | |
| 368 | 389 | if (!dev->mt) { |
| 369 | − /* register with device with a supported NVM manager */ | |
| 370 | − list_for_each_entry(mt, &nvm_mgrs, list) { | |
| 371 | − ret = mt->register_mgr(dev); | |
| 372 | − if (ret < 0) | |
| 373 | − return ret; /* initialization failed */ | |
| 374 | − if (ret > 0) { | |
| 375 | − dev->mt = mt; | |
| 376 | − break; /* successfully initialized */ | |
| 377 | − } | |
| 378 | − } | |
| 379 | − | |
| 380 | − if (!ret) { | |
| 381 | − pr_info("nvm: no compatible nvm manager found.\n"); | |
| 382 | − return -ENODEV; | |
| 383 | − } | |
| 390 | + pr_info("nvm: device has no media manager registered.\n"); | |
| 391 | + return -ENODEV; | |
| 384 | 392 | } |
| 385 | 393 | |
| 394 | + down_write(&nvm_lock); | |
| 386 | 395 | tt = nvm_find_target_type(create->tgttype); |
| 387 | 396 | if (!tt) { |
| 388 | 397 | pr_err("nvm: target type %s not found\n", create->tgttype); |
| 398 | + up_write(&nvm_lock); | |
| 389 | 399 | return -EINVAL; |
| 390 | 400 | } |
| 391 | 401 | |
| 392 | − down_write(&nvm_lock); | |
| 393 | 402 | list_for_each_entry(t, &dev->online_targets, list) { |
| 394 | 403 | if (!strcmp(create->tgtname, t->disk->disk_name)) { |
| 395 | 404 | pr_err("nvm: target name already exists.\n"); |
| @@ -457,11 +466,11 @@ static void nvm_remove_target(struct nvm_target *t) · 61 unmodified lines | ||
| 457 | 466 | lockdep_assert_held(&nvm_lock); |
| 458 | 467 | |
| 459 | 468 | del_gendisk(tdisk); |
| 469 | + blk_cleanup_queue(q); | |
| 470 | + | |
| 460 | 471 | if (tt->exit) |
| 461 | 472 | tt->exit(tdisk->private_data); |
| 462 | 473 | |
| 463 | − blk_cleanup_queue(q); | |
| 464 | − | |
| 465 | 474 | put_disk(tdisk); |
| 466 | 475 | |
| 467 | 476 | list_del(&t->list); |
| @@ -473,7 +482,9 @@ static int __nvm_configure_create(struct nvm_ioctl_create *create) · 5 unmodified lines | ||
| 473 | 482 | struct nvm_dev *dev; |
| 474 | 483 | struct nvm_ioctl_create_simple *s; |
| 475 | 484 | |
| 485 | + down_write(&nvm_lock); | |
| 476 | 486 | dev = nvm_find_nvm_dev(create->dev); |
| 487 | + up_write(&nvm_lock); | |
| 477 | 488 | if (!dev) { |
| 478 | 489 | pr_err("nvm: device not found\n"); |
| 479 | 490 | return -EINVAL; |
| @@ -532,7 +543,9 @@ static int nvm_configure_show(const char *val) · 52 unmodified lines | ||
| 532 | 543 | return -EINVAL; |
| 533 | 544 | } |
| 534 | 545 | |
| 546 | + down_write(&nvm_lock); | |
| 535 | 547 | dev = nvm_find_nvm_dev(devname); |
| 548 | + up_write(&nvm_lock); | |
| 536 | 549 | if (!dev) { |
| 537 | 550 | pr_err("nvm: device not found\n"); |
| 538 | 551 | return -EINVAL; |
| @@ -541,7 +554,7 @@ static int nvm_configure_show(const char *val) · 2 unmodified lines | ||
| 541 | 554 | if (!dev->mt) |
| 542 | 555 | return 0; |
| 543 | 556 | |
| 544 | − dev->mt->free_blocks_print(dev); | |
| 557 | + dev->mt->lun_info_print(dev); | |
| 545 | 558 | |
| 546 | 559 | return 0; |
| 547 | 560 | } |
| @@ -677,8 +690,10 @@ static long nvm_ioctl_info(struct file *file, void __user *arg) · 129 unmodified lines | ||
| 677 | 690 | info->tgtsize = tgt_iter; |
| 678 | 691 | up_write(&nvm_lock); |
| 679 | 692 | |
| 680 | − if (copy_to_user(arg, info, sizeof(struct nvm_ioctl_info))) | |
| 693 | + if (copy_to_user(arg, info, sizeof(struct nvm_ioctl_info))) { | |
| 694 | + kfree(info); | |
| 681 | 695 | return -EFAULT; |
| 696 | + } | |
| 682 | 697 | |
| 683 | 698 | kfree(info); |
| 684 | 699 | return 0; |
| @@ -721,8 +736,11 @@ static long nvm_ioctl_get_devices(struct file *file, void __user *arg) · 36 unmodified lines | ||
| 721 | 736 | |
| 722 | 737 | devices->nr_devices = i; |
| 723 | 738 | |
| 724 | − if (copy_to_user(arg, devices, sizeof(struct nvm_ioctl_get_devices))) | |
| 739 | + if (copy_to_user(arg, devices, | |
| 740 | + sizeof(struct nvm_ioctl_get_devices))) { | |
| 741 | + kfree(devices); | |
| 725 | 742 | return -EFAULT; |
| 743 | + } | |
| 726 | 744 | |
| 727 | 745 | kfree(devices); |
| 728 | 746 | return 0; |
| @@ -60,23 +60,27 @@ static int gennvm_luns_init(struct nvm_dev *dev, struct gen_nvm *gn) · 59 unmodified lines | ||
| 60 | 60 | lun->vlun.lun_id = i % dev->luns_per_chnl; |
| 61 | 61 | lun->vlun.chnl_id = i / dev->luns_per_chnl; |
| 62 | 62 | lun->vlun.nr_free_blocks = dev->blks_per_lun; |
| 63 | + lun->vlun.nr_inuse_blocks = 0; | |
| 64 | + lun->vlun.nr_bad_blocks = 0; | |
| 63 | 65 | } |
| 64 | 66 | return 0; |
| 65 | 67 | } |
| 66 | 68 | |
| 67 | −static int gennvm_block_bb(u32 lun_id, void *bb_bitmap, unsigned int nr_blocks, | |
| 69 | +static int gennvm_block_bb(struct ppa_addr ppa, int nr_blocks, u8 *blks, | |
| 68 | 70 | void *private) |
| 69 | 71 | { |
| 70 | 72 | struct gen_nvm *gn = private; |
| 71 | − struct gen_lun *lun = &gn->luns[lun_id]; | |
| 73 | + struct nvm_dev *dev = gn->dev; | |
| 74 | + struct gen_lun *lun; | |
| 72 | 75 | struct nvm_block *blk; |
| 73 | 76 | int i; |
| 74 | 77 | |
| 75 | − if (unlikely(bitmap_empty(bb_bitmap, nr_blocks))) | |
| 76 | − return 0; | |
| 78 | + lun = &gn->luns[(dev->nr_luns * ppa.g.ch) + ppa.g.lun]; | |
| 79 | + | |
| 80 | + for (i = 0; i < nr_blocks; i++) { | |
| 81 | + if (blks[i] == 0) | |
| 82 | + continue; | |
| 77 | 83 | |
| 78 | − i = -1; | |
| 79 | − while ((i = find_next_bit(bb_bitmap, nr_blocks, i + 1)) < nr_blocks) { | |
| 80 | 84 | blk = &lun->vlun.blocks[i]; |
| 81 | 85 | if (!blk) { |
| 82 | 86 | pr_err("gennvm: BB data is out of bounds.\n"); |
| @@ -84,6 +88,7 @@ static int gennvm_block_bb(u32 lun_id, void *bb_bitmap, unsigned int nr_blocks, · 1 unmodified line | ||
| 84 | 88 | } |
| 85 | 89 | |
| 86 | 90 | list_move_tail(&blk->list, &lun->bb_list); |
| 91 | + lun->vlun.nr_bad_blocks++; | |
| 87 | 92 | } |
| 88 | 93 | |
| 89 | 94 | return 0; |
| @@ -136,6 +141,7 @@ static int gennvm_block_map(u64 slba, u32 nlb, __le64 *entries, void *private) · 46 unmodified lines | ||
| 136 | 141 | list_move_tail(&blk->list, &lun->used_list); |
| 137 | 142 | blk->type = 1; |
| 138 | 143 | lun->vlun.nr_free_blocks--; |
| 144 | + lun->vlun.nr_inuse_blocks++; | |
| 139 | 145 | } |
| 140 | 146 | } |
| 141 | 147 | |
| @@ -164,22 +170,32 @@ static int gennvm_blocks_init(struct nvm_dev *dev, struct gen_nvm *gn) · 22 unmodified lines | ||
| 164 | 170 | block->id = cur_block_id++; |
| 165 | 171 | |
| 166 | 172 | /* First block is reserved for device */ |
| 167 | − if (unlikely(lun_iter == 0 && blk_iter == 0)) | |
| 173 | + if (unlikely(lun_iter == 0 && blk_iter == 0)) { | |
| 174 | + lun->vlun.nr_free_blocks--; | |
| 168 | 175 | continue; |
| 176 | + } | |
| 169 | 177 | |
| 170 | 178 | list_add_tail(&block->list, &lun->free_list); |
| 171 | 179 | } |
| 172 | 180 | |
| 173 | 181 | if (dev->ops->get_bb_tbl) { |
| 174 | − ret = dev->ops->get_bb_tbl(dev->q, lun->vlun.id, | |
| 175 | − dev->blks_per_lun, gennvm_block_bb, gn); | |
| 182 | + struct ppa_addr ppa; | |
| 183 | + | |
| 184 | + ppa.ppa = 0; | |
| 185 | + ppa.g.ch = lun->vlun.chnl_id; | |
| 186 | + ppa.g.lun = lun->vlun.id; | |
| 187 | + ppa = generic_to_dev_addr(dev, ppa); | |
| 188 | + | |
| 189 | + ret = dev->ops->get_bb_tbl(dev, ppa, | |
| 190 | + dev->blks_per_lun, | |
| 191 | + gennvm_block_bb, gn); | |
| 176 | 192 | if (ret) |
| 177 | 193 | pr_err("gennvm: could not read BB table\n"); |
| 178 | 194 | } |
| 179 | 195 | } |
| 180 | 196 | |
| 181 | 197 | if (dev->ops->get_l2p_tbl) { |
| 182 | − ret = dev->ops->get_l2p_tbl(dev->q, 0, dev->total_pages, | |
| 198 | + ret = dev->ops->get_l2p_tbl(dev, 0, dev->total_pages, | |
| 183 | 199 | gennvm_block_map, dev); |
| 184 | 200 | if (ret) { |
| 185 | 201 | pr_err("gennvm: could not read L2P table.\n"); |
| @@ -190,15 +206,27 @@ static int gennvm_blocks_init(struct nvm_dev *dev, struct gen_nvm *gn) · 4 unmodified lines | ||
| 190 | 206 | return 0; |
| 191 | 207 | } |
| 192 | 208 | |
| 209 | +static void gennvm_free(struct nvm_dev *dev) | |
| 210 | +{ | |
| 211 | + gennvm_blocks_free(dev); | |
| 212 | + gennvm_luns_free(dev); | |
| 213 | + kfree(dev->mp); | |
| 214 | + dev->mp = NULL; | |
| 215 | +} | |
| 216 | + | |
| 193 | 217 | static int gennvm_register(struct nvm_dev *dev) |
| 194 | 218 | { |
| 195 | 219 | struct gen_nvm *gn; |
| 196 | 220 | int ret; |
| 197 | 221 | |
| 222 | + if (!try_module_get(THIS_MODULE)) | |
| 223 | + return -ENODEV; | |
| 224 | + | |
| 198 | 225 | gn = kzalloc(sizeof(struct gen_nvm), GFP_KERNEL); |
| 199 | 226 | if (!gn) |
| 200 | 227 | return -ENOMEM; |
| 201 | 228 | |
| 229 | + gn->dev = dev; | |
| 202 | 230 | gn->nr_luns = dev->nr_luns; |
| 203 | 231 | dev->mp = gn; |
| 204 | 232 | |
| @@ -216,16 +244,15 @@ static int gennvm_register(struct nvm_dev *dev) · 11 unmodified lines | ||
| 216 | 244 | |
| 217 | 245 | return 1; |
| 218 | 246 | err: |
| 219 | − kfree(gn); | |
| 247 | + gennvm_free(dev); | |
| 248 | + module_put(THIS_MODULE); | |
| 220 | 249 | return ret; |
| 221 | 250 | } |
| 222 | 251 | |
| 223 | 252 | static void gennvm_unregister(struct nvm_dev *dev) |
| 224 | 253 | { |
| 225 | − gennvm_blocks_free(dev); | |
| 226 | − gennvm_luns_free(dev); | |
| 227 | − kfree(dev->mp); | |
| 228 | − dev->mp = NULL; | |
| 254 | + gennvm_free(dev); | |
| 255 | + module_put(THIS_MODULE); | |
| 229 | 256 | } |
| 230 | 257 | |
| 231 | 258 | static struct nvm_block *gennvm_get_blk(struct nvm_dev *dev, |
| @@ -240,23 +267,21 @@ static struct nvm_block *gennvm_get_blk(struct nvm_dev *dev, · 8 unmodified lines | ||
| 240 | 267 | if (list_empty(&lun->free_list)) { |
| 241 | 268 | pr_err_ratelimited("gennvm: lun %u have no free pages available", |
| 242 | 269 | lun->vlun.id); |
| 243 | − spin_unlock(&vlun->lock); | |
| 244 | 270 | goto out; |
| 245 | 271 | } |
| 246 | 272 | |
| 247 | − while (!is_gc && lun->vlun.nr_free_blocks < lun->reserved_blocks) { | |
| 248 | − spin_unlock(&vlun->lock); | |
| 273 | + if (!is_gc && lun->vlun.nr_free_blocks < lun->reserved_blocks) | |
| 249 | 274 | goto out; |
| 250 | − } | |
| 251 | 275 | |
| 252 | 276 | blk = list_first_entry(&lun->free_list, struct nvm_block, list); |
| 253 | 277 | list_move_tail(&blk->list, &lun->used_list); |
| 254 | 278 | blk->type = 1; |
| 255 | 279 | |
| 256 | 280 | lun->vlun.nr_free_blocks--; |
| 281 | + lun->vlun.nr_inuse_blocks++; | |
| 257 | 282 | |
| 258 | − spin_unlock(&vlun->lock); | |
| 259 | 283 | out: |
| 284 | + spin_unlock(&vlun->lock); | |
| 260 | 285 | return blk; |
| 261 | 286 | } |
| 262 | 287 | |
| @@ -271,16 +296,21 @@ static void gennvm_put_blk(struct nvm_dev *dev, struct nvm_block *blk) · 8 unmodified lines | ||
| 271 | 296 | case 1: |
| 272 | 297 | list_move_tail(&blk->list, &lun->free_list); |
| 273 | 298 | lun->vlun.nr_free_blocks++; |
| 299 | + lun->vlun.nr_inuse_blocks--; | |
| 274 | 300 | blk->type = 0; |
| 275 | 301 | break; |
| 276 | 302 | case 2: |
| 277 | 303 | list_move_tail(&blk->list, &lun->bb_list); |
| 304 | + lun->vlun.nr_bad_blocks++; | |
| 305 | + lun->vlun.nr_inuse_blocks--; | |
| 278 | 306 | break; |
| 279 | 307 | default: |
| 280 | 308 | WARN_ON_ONCE(1); |
| 281 | 309 | pr_err("gennvm: erroneous block type (%lu -> %u)\n", |
| 282 | 310 | blk->id, blk->type); |
| 283 | 311 | list_move_tail(&blk->list, &lun->bb_list); |
| 312 | + lun->vlun.nr_bad_blocks++; | |
| 313 | + lun->vlun.nr_inuse_blocks--; | |
| 284 | 314 | } |
| 285 | 315 | |
| 286 | 316 | spin_unlock(&vlun->lock); |
| @@ -292,10 +322,10 @@ static void gennvm_addr_to_generic_mode(struct nvm_dev *dev, struct nvm_rq *rqd) · 5 unmodified lines | ||
| 292 | 322 | |
| 293 | 323 | if (rqd->nr_pages > 1) { |
| 294 | 324 | for (i = 0; i < rqd->nr_pages; i++) |
| 295 | − rqd->ppa_list[i] = addr_to_generic_mode(dev, | |
| 325 | + rqd->ppa_list[i] = dev_to_generic_addr(dev, | |
| 296 | 326 | rqd->ppa_list[i]); |
| 297 | 327 | } else { |
| 298 | − rqd->ppa_addr = addr_to_generic_mode(dev, rqd->ppa_addr); | |
| 328 | + rqd->ppa_addr = dev_to_generic_addr(dev, rqd->ppa_addr); | |
| 299 | 329 | } |
| 300 | 330 | } |
| 301 | 331 | |
| @@ -305,10 +335,10 @@ static void gennvm_generic_to_addr_mode(struct nvm_dev *dev, struct nvm_rq *rqd) · 3 unmodified lines | ||
| 305 | 335 | |
| 306 | 336 | if (rqd->nr_pages > 1) { |
| 307 | 337 | for (i = 0; i < rqd->nr_pages; i++) |
| 308 | − rqd->ppa_list[i] = generic_to_addr_mode(dev, | |
| 338 | + rqd->ppa_list[i] = generic_to_dev_addr(dev, | |
| 309 | 339 | rqd->ppa_list[i]); |
| 310 | 340 | } else { |
| 311 | − rqd->ppa_addr = generic_to_addr_mode(dev, rqd->ppa_addr); | |
| 341 | + rqd->ppa_addr = generic_to_dev_addr(dev, rqd->ppa_addr); | |
| 312 | 342 | } |
| 313 | 343 | } |
| 314 | 344 | |
| @@ -321,7 +351,7 @@ static int gennvm_submit_io(struct nvm_dev *dev, struct nvm_rq *rqd) · 6 unmodified lines | ||
| 321 | 351 | gennvm_generic_to_addr_mode(dev, rqd); |
| 322 | 352 | |
| 323 | 353 | rqd->dev = dev; |
| 324 | − return dev->ops->submit_io(dev->q, rqd); | |
| 354 | + return dev->ops->submit_io(dev, rqd); | |
| 325 | 355 | } |
| 326 | 356 | |
| 327 | 357 | static void gennvm_blk_set_type(struct nvm_dev *dev, struct ppa_addr *ppa, |
| @@ -354,10 +384,10 @@ static void gennvm_mark_blk_bad(struct nvm_dev *dev, struct nvm_rq *rqd) · 26 unmodified lines | ||
| 354 | 384 | { |
| 355 | 385 | int i; |
| 356 | 386 | |
| 357 | − if (!dev->ops->set_bb) | |
| 387 | + if (!dev->ops->set_bb_tbl) | |
| 358 | 388 | return; |
| 359 | 389 | |
| 360 | − if (dev->ops->set_bb(dev->q, rqd, 1)) | |
| 390 | + if (dev->ops->set_bb_tbl(dev, rqd, 1)) | |
| 361 | 391 | return; |
| 362 | 392 | |
| 363 | 393 | gennvm_addr_to_generic_mode(dev, rqd); |
| @@ -425,7 +455,7 @@ static int gennvm_erase_blk(struct nvm_dev *dev, struct nvm_block *blk, · 61 unmodified lines | ||
| 425 | 455 | |
| 426 | 456 | gennvm_generic_to_addr_mode(dev, &rqd); |
| 427 | 457 | |
| 428 | − ret = dev->ops->erase_block(dev->q, &rqd); | |
| 458 | + ret = dev->ops->erase_block(dev, &rqd); | |
| 429 | 459 | |
| 430 | 460 | if (plane_cnt) |
| 431 | 461 | nvm_dev_dma_free(dev, rqd.ppa_list, rqd.dma_ppa_list); |
| @@ -440,15 +470,24 @@ static struct nvm_lun *gennvm_get_lun(struct nvm_dev *dev, int lunid) · 8 unmodified lines | ||
| 440 | 470 | return &gn->luns[lunid].vlun; |
| 441 | 471 | } |
| 442 | 472 | |
| 443 | −static void gennvm_free_blocks_print(struct nvm_dev *dev) | |
| 473 | +static void gennvm_lun_info_print(struct nvm_dev *dev) | |
| 444 | 474 | { |
| 445 | 475 | struct gen_nvm *gn = dev->mp; |
| 446 | 476 | struct gen_lun *lun; |
| 447 | 477 | unsigned int i; |
| 448 | 478 | |
| 449 | − gennvm_for_each_lun(gn, lun, i) | |
| 450 | − pr_info("%s: lun%8u\t%u\n", | |
| 451 | − dev->name, i, lun->vlun.nr_free_blocks); | |
| 479 | + | |
| 480 | + gennvm_for_each_lun(gn, lun, i) { | |
| 481 | + spin_lock(&lun->vlun.lock); | |
| 482 | + | |
| 483 | + pr_info("%s: lun%8u\t%u\t%u\t%u\n", | |
| 484 | + dev->name, i, | |
| 485 | + lun->vlun.nr_free_blocks, | |
| 486 | + lun->vlun.nr_inuse_blocks, | |
| 487 | + lun->vlun.nr_bad_blocks); | |
| 488 | + | |
| 489 | + spin_unlock(&lun->vlun.lock); | |
| 490 | + } | |
| 452 | 491 | } |
| 453 | 492 | |
| 454 | 493 | static struct nvmm_type gennvm = { |
| @@ -466,7 +505,7 @@ static struct nvmm_type gennvm = { · 11 unmodified lines | ||
| 466 | 505 | .erase_blk = gennvm_erase_blk, |
| 467 | 506 | |
| 468 | 507 | .get_lun = gennvm_get_lun, |
| 469 | − .free_blocks_print = gennvm_free_blocks_print, | |
| 508 | + .lun_info_print = gennvm_lun_info_print, | |
| 470 | 509 | }; |
| 471 | 510 | |
| 472 | 511 | static int __init gennvm_module_init(void) |
| @@ -35,6 +35,8 @@ struct gen_lun { · 34 unmodified lines | ||
| 35 | 35 | }; |
| 36 | 36 | |
| 37 | 37 | struct gen_nvm { |
| 38 | + struct nvm_dev *dev; | |
| 39 | + | |
| 38 | 40 | int nr_luns; |
| 39 | 41 | struct gen_lun *luns; |
| 40 | 42 | }; |
| @@ -123,12 +123,42 @@ static u64 block_to_addr(struct rrpc *rrpc, struct rrpc_block *rblk) · 122 unmodified lines | ||
| 123 | 123 | return blk->id * rrpc->dev->pgs_per_blk; |
| 124 | 124 | } |
| 125 | 125 | |
| 126 | +static struct ppa_addr linear_to_generic_addr(struct nvm_dev *dev, | |
| 127 | + struct ppa_addr r) | |
| 128 | +{ | |
| 129 | + struct ppa_addr l; | |
| 130 | + int secs, pgs, blks, luns; | |
| 131 | + sector_t ppa = r.ppa; | |
| 132 | + | |
| 133 | + l.ppa = 0; | |
| 134 | + | |
| 135 | + div_u64_rem(ppa, dev->sec_per_pg, &secs); | |
| 136 | + l.g.sec = secs; | |
| 137 | + | |
| 138 | + sector_div(ppa, dev->sec_per_pg); | |
| 139 | + div_u64_rem(ppa, dev->sec_per_blk, &pgs); | |
| 140 | + l.g.pg = pgs; | |
| 141 | + | |
| 142 | + sector_div(ppa, dev->pgs_per_blk); | |
| 143 | + div_u64_rem(ppa, dev->blks_per_lun, &blks); | |
| 144 | + l.g.blk = blks; | |
| 145 | + | |
| 146 | + sector_div(ppa, dev->blks_per_lun); | |
| 147 | + div_u64_rem(ppa, dev->luns_per_chnl, &luns); | |
| 148 | + l.g.lun = luns; | |
| 149 | + | |
| 150 | + sector_div(ppa, dev->luns_per_chnl); | |
| 151 | + l.g.ch = ppa; | |
| 152 | + | |
| 153 | + return l; | |
| 154 | +} | |
| 155 | + | |
| 126 | 156 | static struct ppa_addr rrpc_ppa_to_gaddr(struct nvm_dev *dev, u64 addr) |
| 127 | 157 | { |
| 128 | 158 | struct ppa_addr paddr; |
| 129 | 159 | |
| 130 | 160 | paddr.ppa = addr; |
| 131 | − return __linear_to_generic_addr(dev, paddr); | |
| 161 | + return linear_to_generic_addr(dev, paddr); | |
| 132 | 162 | } |
| 133 | 163 | |
| 134 | 164 | /* requires lun->lock taken */ |
| @@ -152,7 +182,7 @@ static struct rrpc_block *rrpc_get_blk(struct rrpc *rrpc, struct rrpc_lun *rlun, · 17 unmodified lines | ||
| 152 | 182 | struct nvm_block *blk; |
| 153 | 183 | struct rrpc_block *rblk; |
| 154 | 184 | |
| 155 | − blk = nvm_get_blk(rrpc->dev, rlun->parent, 0); | |
| 185 | + blk = nvm_get_blk(rrpc->dev, rlun->parent, flags); | |
| 156 | 186 | if (!blk) |
| 157 | 187 | return NULL; |
| 158 | 188 | |
| @@ -172,6 +202,20 @@ static void rrpc_put_blk(struct rrpc *rrpc, struct rrpc_block *rblk) · 13 unmodified lines | ||
| 172 | 202 | nvm_put_blk(rrpc->dev, rblk->parent); |
| 173 | 203 | } |
| 174 | 204 | |
| 205 | +static void rrpc_put_blks(struct rrpc *rrpc) | |
| 206 | +{ | |
| 207 | + struct rrpc_lun *rlun; | |
| 208 | + int i; | |
| 209 | + | |
| 210 | + for (i = 0; i < rrpc->nr_luns; i++) { | |
| 211 | + rlun = &rrpc->luns[i]; | |
| 212 | + if (rlun->cur) | |
| 213 | + rrpc_put_blk(rrpc, rlun->cur); | |
| 214 | + if (rlun->gc_cur) | |
| 215 | + rrpc_put_blk(rrpc, rlun->gc_cur); | |
| 216 | + } | |
| 217 | +} | |
| 218 | + | |
| 175 | 219 | static struct rrpc_lun *get_next_lun(struct rrpc *rrpc) |
| 176 | 220 | { |
| 177 | 221 | int next = atomic_inc_return(&rrpc->next_lun); |
| @@ -972,7 +1016,7 @@ static int rrpc_map_init(struct rrpc *rrpc) · 794 unmodified lines | ||
| 972 | 1016 | return 0; |
| 973 | 1017 | |
| 974 | 1018 | /* Bring up the mapping table from device */ |
| 975 | − ret = dev->ops->get_l2p_tbl(dev->q, 0, dev->total_pages, | |
| 1019 | + ret = dev->ops->get_l2p_tbl(dev, 0, dev->total_pages, | |
| 976 | 1020 | rrpc_l2p_update, rrpc); |
| 977 | 1021 | if (ret) { |
| 978 | 1022 | pr_err("nvm: rrpc: could not read L2P table.\n"); |
| @@ -1194,18 +1238,21 @@ static int rrpc_luns_configure(struct rrpc *rrpc) · 215 unmodified lines | ||
| 1194 | 1238 | |
| 1195 | 1239 | rblk = rrpc_get_blk(rrpc, rlun, 0); |
| 1196 | 1240 | if (!rblk) |
| 1197 | − return -EINVAL; | |
| 1241 | + goto err; | |
| 1198 | 1242 | |
| 1199 | 1243 | rrpc_set_lun_cur(rlun, rblk); |
| 1200 | 1244 | |
| 1201 | 1245 | /* Emergency gc block */ |
| 1202 | 1246 | rblk = rrpc_get_blk(rrpc, rlun, 1); |
| 1203 | 1247 | if (!rblk) |
| 1204 | − return -EINVAL; | |
| 1248 | + goto err; | |
| 1205 | 1249 | rlun->gc_cur = rblk; |
| 1206 | 1250 | } |
| 1207 | 1251 | |
| 1208 | 1252 | return 0; |
| 1253 | +err: | |
| 1254 | + rrpc_put_blks(rrpc); | |
| 1255 | + return -EINVAL; | |
| 1209 | 1256 | } |
| 1210 | 1257 | |
| 1211 | 1258 | static struct nvm_tgt_type tt_rrpc; |
| @@ -112,7 +112,8 @@ struct iv_tcw_private { · 111 unmodified lines | ||
| 112 | 112 | * and encrypts / decrypts at the same time. |
| 113 | 113 | */ |
| 114 | 114 | enum flags { DM_CRYPT_SUSPENDED, DM_CRYPT_KEY_VALID, |
| 115 | − DM_CRYPT_SAME_CPU, DM_CRYPT_NO_OFFLOAD }; | |
| 115 | + DM_CRYPT_SAME_CPU, DM_CRYPT_NO_OFFLOAD, | |
| 116 | + DM_CRYPT_EXIT_THREAD}; | |
| 116 | 117 | |
| 117 | 118 | /* |
| 118 | 119 | * The fields in here must be read only after initialization. |
| @@ -1203,20 +1204,18 @@ continue_locked: · 1084 unmodified lines | ||
| 1203 | 1204 | if (!RB_EMPTY_ROOT(&cc->write_tree)) |
| 1204 | 1205 | goto pop_from_list; |
| 1205 | 1206 | |
| 1207 | + if (unlikely(test_bit(DM_CRYPT_EXIT_THREAD, &cc->flags))) { | |
| 1208 | + spin_unlock_irq(&cc->write_thread_wait.lock); | |
| 1209 | + break; | |
| 1210 | + } | |
| 1211 | + | |
| 1206 | 1212 | __set_current_state(TASK_INTERRUPTIBLE); |
| 1207 | 1213 | __add_wait_queue(&cc->write_thread_wait, &wait); |
| 1208 | 1214 | |
| 1209 | 1215 | spin_unlock_irq(&cc->write_thread_wait.lock); |
| 1210 | 1216 | |
| 1211 | − if (unlikely(kthread_should_stop())) { | |
| 1212 | − set_task_state(current, TASK_RUNNING); | |
| 1213 | − remove_wait_queue(&cc->write_thread_wait, &wait); | |
| 1214 | − break; | |
| 1215 | − } | |
| 1216 | − | |
| 1217 | 1217 | schedule(); |
| 1218 | 1218 | |
| 1219 | − set_task_state(current, TASK_RUNNING); | |
| 1220 | 1219 | spin_lock_irq(&cc->write_thread_wait.lock); |
| 1221 | 1220 | __remove_wait_queue(&cc->write_thread_wait, &wait); |
| 1222 | 1221 | goto continue_locked; |
| @@ -1531,8 +1530,13 @@ static void crypt_dtr(struct dm_target *ti) · 308 unmodified lines | ||
| 1531 | 1530 | if (!cc) |
| 1532 | 1531 | return; |
| 1533 | 1532 | |
| 1534 | − if (cc->write_thread) | |
| 1533 | + if (cc->write_thread) { | |
| 1534 | + spin_lock_irq(&cc->write_thread_wait.lock); | |
| 1535 | + set_bit(DM_CRYPT_EXIT_THREAD, &cc->flags); | |
| 1536 | + wake_up_locked(&cc->write_thread_wait); | |
| 1537 | + spin_unlock_irq(&cc->write_thread_wait.lock); | |
| 1535 | 1538 | kthread_stop(cc->write_thread); |
| 1539 | + } | |
| 1536 | 1540 | |
| 1537 | 1541 | if (cc->io_queue) |
| 1538 | 1542 | destroy_workqueue(cc->io_queue); |
| @@ -1537,32 +1537,34 @@ static int multipath_prepare_ioctl(struct dm_target *ti, · 1536 unmodified lines | ||
| 1537 | 1537 | struct block_device **bdev, fmode_t *mode) |
| 1538 | 1538 | { |
| 1539 | 1539 | struct multipath *m = ti->private; |
| 1540 | − struct pgpath *pgpath; | |
| 1541 | 1540 | unsigned long flags; |
| 1542 | 1541 | int r; |
| 1543 | 1542 | |
| 1544 | − r = 0; | |
| 1545 | − | |
| 1546 | 1543 | spin_lock_irqsave(&m->lock, flags); |
| 1547 | 1544 | |
| 1548 | 1545 | if (!m->current_pgpath) |
| 1549 | 1546 | __choose_pgpath(m, 0); |
| 1550 | 1547 | |
| 1551 | − pgpath = m->current_pgpath; | |
| 1552 | − | |
| 1553 | − if (pgpath) { | |
| 1554 | − *bdev = pgpath->path.dev->bdev; | |
| 1555 | − *mode = pgpath->path.dev->mode; | |
| 1548 | + if (m->current_pgpath) { | |
| 1549 | + if (!m->queue_io) { | |
| 1550 | + *bdev = m->current_pgpath->path.dev->bdev; | |
| 1551 | + *mode = m->current_pgpath->path.dev->mode; | |
| 1552 | + r = 0; | |
| 1553 | + } else { | |
| 1554 | + /* pg_init has not started or completed */ | |
| 1555 | + r = -ENOTCONN; | |
| 1556 | + } | |
| 1557 | + } else { | |
| 1558 | + /* No path is available */ | |
| 1559 | + if (m->queue_if_no_path) | |
| 1560 | + r = -ENOTCONN; | |
| 1561 | + else | |
| 1562 | + r = -EIO; | |
| 1556 | 1563 | } |
| 1557 | 1564 | |
| 1558 | − if ((pgpath && m->queue_io) || (!pgpath && m->queue_if_no_path)) | |
| 1559 | − r = -ENOTCONN; | |
| 1560 | − else if (!*bdev) | |
| 1561 | − r = -EIO; | |
| 1562 | − | |
| 1563 | 1565 | spin_unlock_irqrestore(&m->lock, flags); |
| 1564 | 1566 | |
| 1565 | − if (r == -ENOTCONN && !fatal_signal_pending(current)) { | |
| 1567 | + if (r == -ENOTCONN) { | |
| 1566 | 1568 | spin_lock_irqsave(&m->lock, flags); |
| 1567 | 1569 | if (!m->current_pg) { |
| 1568 | 1570 | /* Path status changed, redo selection */ |
| @@ -1206,6 +1206,12 @@ static int __reserve_metadata_snap(struct dm_pool_metadata *pmd) · 1205 unmodified lines | ||
| 1206 | 1206 | struct dm_block *copy, *sblock; |
| 1207 | 1207 | dm_block_t held_root; |
| 1208 | 1208 | |
| 1209 | + /* | |
| 1210 | + * We commit to ensure the btree roots which we increment in a | |
| 1211 | + * moment are up to date. | |
| 1212 | + */ | |
| 1213 | + __commit_transaction(pmd); | |
| 1214 | + | |
| 1209 | 1215 | /* |
| 1210 | 1216 | * Copy the superblock. |
| 1211 | 1217 | */ |
| @@ -1538,7 +1544,7 @@ static int __remove(struct dm_thin_device *td, dm_block_t block) · 326 unmodified lines | ||
| 1538 | 1544 | static int __remove_range(struct dm_thin_device *td, dm_block_t begin, dm_block_t end) |
| 1539 | 1545 | { |
| 1540 | 1546 | int r; |
| 1541 | − unsigned count; | |
| 1547 | + unsigned count, total_count = 0; | |
| 1542 | 1548 | struct dm_pool_metadata *pmd = td->pmd; |
| 1543 | 1549 | dm_block_t keys[1] = { td->id }; |
| 1544 | 1550 | __le64 value; |
| @@ -1561,11 +1567,29 @@ static int __remove_range(struct dm_thin_device *td, dm_block_t begin, dm_block_ · 16 unmodified lines | ||
| 1561 | 1567 | if (r) |
| 1562 | 1568 | return r; |
| 1563 | 1569 | |
| 1564 | − r = dm_btree_remove_leaves(&pmd->bl_info, mapping_root, &begin, end, &mapping_root, &count); | |
| 1565 | − if (r) | |
| 1566 | − return r; | |
| 1570 | + /* | |
| 1571 | + * Remove leaves stops at the first unmapped entry, so we have to | |
| 1572 | + * loop round finding mapped ranges. | |
| 1573 | + */ | |
| 1574 | + while (begin < end) { | |
| 1575 | + r = dm_btree_lookup_next(&pmd->bl_info, mapping_root, &begin, &begin, &value); | |
| 1576 | + if (r == -ENODATA) | |
| 1577 | + break; | |
| 1578 | + | |
| 1579 | + if (r) | |
| 1580 | + return r; | |
| 1581 | + | |
| 1582 | + if (begin >= end) | |
| 1583 | + break; | |
| 1584 | + | |
| 1585 | + r = dm_btree_remove_leaves(&pmd->bl_info, mapping_root, &begin, end, &mapping_root, &count); | |
| 1586 | + if (r) | |
| 1587 | + return r; | |
| 1588 | + | |
| 1589 | + total_count += count; | |
| 1590 | + } | |
| 1567 | 1591 | |
| 1568 | − td->mapped_blocks -= count; | |
| 1592 | + td->mapped_blocks -= total_count; | |
| 1569 | 1593 | td->changed = 1; |
| 1570 | 1594 | |
| 1571 | 1595 | /* |
| @@ -2432,6 +2432,7 @@ static void set_pool_mode(struct pool *pool, enum pool_mode new_mode) · 2431 unmodified lines | ||
| 2432 | 2432 | case PM_WRITE: |
| 2433 | 2433 | if (old_mode != new_mode) |
| 2434 | 2434 | notify_of_pool_mode_change(pool, "write"); |
| 2435 | + pool->pf.error_if_no_space = pt->requested_pf.error_if_no_space; | |
| 2435 | 2436 | dm_pool_metadata_read_write(pool->pmd); |
| 2436 | 2437 | pool->process_bio = process_bio; |
| 2437 | 2438 | pool->process_discard = process_discard_bio; |
| @@ -4249,10 +4250,9 @@ static void thin_io_hints(struct dm_target *ti, struct queue_limits *limits) · 1811 unmodified lines | ||
| 4249 | 4250 | { |
| 4250 | 4251 | struct thin_c *tc = ti->private; |
| 4251 | 4252 | struct pool *pool = tc->pool; |
| 4252 | − struct queue_limits *pool_limits = dm_get_queue_limits(pool->pool_md); | |
| 4253 | 4253 | |
| 4254 | − if (!pool_limits->discard_granularity) | |
| 4255 | − return; /* pool's discard support is disabled */ | |
| 4254 | + if (!pool->pf.discard_enabled) | |
| 4255 | + return; | |
| 4256 | 4256 | |
| 4257 | 4257 | limits->discard_granularity = pool->sectors_per_block << SECTOR_SHIFT; |
| 4258 | 4258 | limits->max_discard_sectors = 2048 * 1024 * 16; /* 16G */ |
| @@ -591,7 +591,7 @@ retry: · 590 unmodified lines | ||
| 591 | 591 | |
| 592 | 592 | out: |
| 593 | 593 | dm_put_live_table(md, *srcu_idx); |
| 594 | − if (r == -ENOTCONN) { | |
| 594 | + if (r == -ENOTCONN && !fatal_signal_pending(current)) { | |
| 595 | 595 | msleep(10); |
| 596 | 596 | goto retry; |
| 597 | 597 | } |
| @@ -603,9 +603,10 @@ static int dm_blk_ioctl(struct block_device *bdev, fmode_t mode, · 5 unmodified lines | ||
| 603 | 603 | { |
| 604 | 604 | struct mapped_device *md = bdev->bd_disk->private_data; |
| 605 | 605 | struct dm_target *tgt; |
| 606 | + struct block_device *tgt_bdev = NULL; | |
| 606 | 607 | int srcu_idx, r; |
| 607 | 608 | |
| 608 | − r = dm_get_live_table_for_ioctl(md, &tgt, &bdev, &mode, &srcu_idx); | |
| 609 | + r = dm_get_live_table_for_ioctl(md, &tgt, &tgt_bdev, &mode, &srcu_idx); | |
| 609 | 610 | if (r < 0) |
| 610 | 611 | return r; |
| 611 | 612 | |
| @@ -620,7 +621,7 @@ static int dm_blk_ioctl(struct block_device *bdev, fmode_t mode, · 8 unmodified lines | ||
| 620 | 621 | goto out; |
| 621 | 622 | } |
| 622 | 623 | |
| 623 | − r = __blkdev_driver_ioctl(bdev, mode, cmd, arg); | |
| 624 | + r = __blkdev_driver_ioctl(tgt_bdev, mode, cmd, arg); | |
| 624 | 625 | out: |
| 625 | 626 | dm_put_live_table(md, srcu_idx); |
| 626 | 627 | return r; |
| @@ -63,6 +63,11 @@ int lower_bound(struct btree_node *n, uint64_t key) · 62 unmodified lines | ||
| 63 | 63 | return bsearch(n, key, 0); |
| 64 | 64 | } |
| 65 | 65 | |
| 66 | +static int upper_bound(struct btree_node *n, uint64_t key) | |
| 67 | +{ | |
| 68 | + return bsearch(n, key, 1); | |
| 69 | +} | |
| 70 | + | |
| 66 | 71 | void inc_children(struct dm_transaction_manager *tm, struct btree_node *n, |
| 67 | 72 | struct dm_btree_value_type *vt) |
| 68 | 73 | { |
| @@ -252,6 +257,16 @@ static void pop_frame(struct del_stack *s) · 183 unmodified lines | ||
| 252 | 257 | dm_tm_unlock(s->tm, f->b); |
| 253 | 258 | } |
| 254 | 259 | |
| 260 | +static void unlock_all_frames(struct del_stack *s) | |
| 261 | +{ | |
| 262 | + struct frame *f; | |
| 263 | + | |
| 264 | + while (unprocessed_frames(s)) { | |
| 265 | + f = s->spine + s->top--; | |
| 266 | + dm_tm_unlock(s->tm, f->b); | |
| 267 | + } | |
| 268 | +} | |
| 269 | + | |
| 255 | 270 | int dm_btree_del(struct dm_btree_info *info, dm_block_t root) |
| 256 | 271 | { |
| 257 | 272 | int r; |
| @@ -308,9 +323,13 @@ int dm_btree_del(struct dm_btree_info *info, dm_block_t root) · 50 unmodified lines | ||
| 308 | 323 | pop_frame(s); |
| 309 | 324 | } |
| 310 | 325 | } |
| 311 | − | |
| 312 | 326 | out: |
| 327 | + if (r) { | |
| 328 | + /* cleanup all frames of del_stack */ | |
| 329 | + unlock_all_frames(s); | |
| 330 | + } | |
| 313 | 331 | kfree(s); |
| 332 | + | |
| 314 | 333 | return r; |
| 315 | 334 | } |
| 316 | 335 | EXPORT_SYMBOL_GPL(dm_btree_del); |
| @@ -392,6 +411,82 @@ int dm_btree_lookup(struct dm_btree_info *info, dm_block_t root, · 75 unmodified lines | ||
| 392 | 411 | } |
| 393 | 412 | EXPORT_SYMBOL_GPL(dm_btree_lookup); |
| 394 | 413 | |
| 414 | +static int dm_btree_lookup_next_single(struct dm_btree_info *info, dm_block_t root, | |
| 415 | + uint64_t key, uint64_t *rkey, void *value_le) | |
| 416 | +{ | |
| 417 | + int r, i; | |
| 418 | + uint32_t flags, nr_entries; | |
| 419 | + struct dm_block *node; | |
| 420 | + struct btree_node *n; | |
| 421 | + | |
| 422 | + r = bn_read_lock(info, root, &node); | |
| 423 | + if (r) | |
| 424 | + return r; | |
| 425 | + | |
| 426 | + n = dm_block_data(node); | |
| 427 | + flags = le32_to_cpu(n->header.flags); | |
| 428 | + nr_entries = le32_to_cpu(n->header.nr_entries); | |
| 429 | + | |
| 430 | + if (flags & INTERNAL_NODE) { | |
| 431 | + i = lower_bound(n, key); | |
| 432 | + if (i < 0 || i >= nr_entries) { | |
| 433 | + r = -ENODATA; | |
| 434 | + goto out; | |
| 435 | + } | |
| 436 | + | |
| 437 | + r = dm_btree_lookup_next_single(info, value64(n, i), key, rkey, value_le); | |
| 438 | + if (r == -ENODATA && i < (nr_entries - 1)) { | |
| 439 | + i++; | |
| 440 | + r = dm_btree_lookup_next_single(info, value64(n, i), key, rkey, value_le); | |
| 441 | + } | |
| 442 | + | |
| 443 | + } else { | |
| 444 | + i = upper_bound(n, key); | |
| 445 | + if (i < 0 || i >= nr_entries) { | |
| 446 | + r = -ENODATA; | |
| 447 | + goto out; | |
| 448 | + } | |
| 449 | + | |
| 450 | + *rkey = le64_to_cpu(n->keys[i]); | |
| 451 | + memcpy(value_le, value_ptr(n, i), info->value_type.size); | |
| 452 | + } | |
| 453 | +out: | |
| 454 | + dm_tm_unlock(info->tm, node); | |
| 455 | + return r; | |
| 456 | +} | |
| 457 | + | |
| 458 | +int dm_btree_lookup_next(struct dm_btree_info *info, dm_block_t root, | |
| 459 | + uint64_t *keys, uint64_t *rkey, void *value_le) | |
| 460 | +{ | |
| 461 | + unsigned level; | |
| 462 | + int r = -ENODATA; | |
| 463 | + __le64 internal_value_le; | |
| 464 | + struct ro_spine spine; | |
| 465 | + | |
| 466 | + init_ro_spine(&spine, info); | |
| 467 | + for (level = 0; level < info->levels - 1u; level++) { | |
| 468 | + r = btree_lookup_raw(&spine, root, keys[level], | |
| 469 | + lower_bound, rkey, | |
| 470 | + &internal_value_le, sizeof(uint64_t)); | |
| 471 | + if (r) | |
| 472 | + goto out; | |
| 473 | + | |
| 474 | + if (*rkey != keys[level]) { | |
| 475 | + r = -ENODATA; | |
| 476 | + goto out; | |
| 477 | + } | |
| 478 | + | |
| 479 | + root = le64_to_cpu(internal_value_le); | |
| 480 | + } | |
| 481 | + | |
| 482 | + r = dm_btree_lookup_next_single(info, root, keys[level], rkey, value_le); | |
| 483 | +out: | |
| 484 | + exit_ro_spine(&spine); | |
| 485 | + return r; | |
| 486 | +} | |
| 487 | + | |
| 488 | +EXPORT_SYMBOL_GPL(dm_btree_lookup_next); | |
| 489 | + | |
| 395 | 490 | /* |
| 396 | 491 | * Splits a node by creating a sibling node and shifting half the nodes |
| 397 | 492 | * contents across. Assumes there is a parent node, and it has room for |
| @@ -473,8 +568,10 @@ static int btree_split_sibling(struct shadow_spine *s, unsigned parent_index, · 75 unmodified lines | ||
| 473 | 568 | |
| 474 | 569 | r = insert_at(sizeof(__le64), pn, parent_index + 1, |
| 475 | 570 | le64_to_cpu(rn->keys[0]), &location); |
| 476 | − if (r) | |
| 571 | + if (r) { | |
| 572 | + unlock_block(s->info, right); | |
| 477 | 573 | return r; |
| 574 | + } | |
| 478 | 575 | |
| 479 | 576 | if (key < le64_to_cpu(rn->keys[0])) { |
| 480 | 577 | unlock_block(s->info, right); |
| @@ -109,6 +109,13 @@ int dm_btree_del(struct dm_btree_info *info, dm_block_t root); · 108 unmodified lines | ||
| 109 | 109 | int dm_btree_lookup(struct dm_btree_info *info, dm_block_t root, |
| 110 | 110 | uint64_t *keys, void *value_le); |
| 111 | 111 | |
| 112 | +/* | |
| 113 | + * Tries to find the first key where the bottom level key is >= to that | |
| 114 | + * given. Useful for skipping empty sections of the btree. | |
| 115 | + */ | |
| 116 | +int dm_btree_lookup_next(struct dm_btree_info *info, dm_block_t root, | |
| 117 | + uint64_t *keys, uint64_t *rkey, void *value_le); | |
| 118 | + | |
| 112 | 119 | /* |
| 113 | 120 | * Insertion (or overwrite an existing value). O(ln(n)) |
| 114 | 121 | */ |
| @@ -135,9 +142,10 @@ int dm_btree_remove(struct dm_btree_info *info, dm_block_t root, · 20 unmodified lines | ||
| 135 | 142 | uint64_t *keys, dm_block_t *new_root); |
| 136 | 143 | |
| 137 | 144 | /* |
| 138 | − * Removes values between 'keys' and keys2, where keys2 is keys with the | |
| 139 | − * final key replaced with 'end_key'. 'end_key' is the one-past-the-end | |
| 140 | − * value. 'keys' may be altered. | |
| 145 | + * Removes a _contiguous_ run of values starting from 'keys' and not | |
| 146 | + * reaching keys2 (where keys2 is keys with the final key replaced with | |
| 147 | + * 'end_key'). 'end_key' is the one-past-the-end value. 'keys' may be | |
| 148 | + * altered. | |
| 141 | 149 | */ |
| 142 | 150 | int dm_btree_remove_leaves(struct dm_btree_info *info, dm_block_t root, |
| 143 | 151 | uint64_t *keys, uint64_t end_key, |
| @@ -136,7 +136,7 @@ static int brb_push(struct bop_ring_buffer *brb, · 135 unmodified lines | ||
| 136 | 136 | return 0; |
| 137 | 137 | } |
| 138 | 138 | |
| 139 | −static int brb_pop(struct bop_ring_buffer *brb, struct block_op *result) | |
| 139 | +static int brb_peek(struct bop_ring_buffer *brb, struct block_op *result) | |
| 140 | 140 | { |
| 141 | 141 | struct block_op *bop; |
| 142 | 142 | |
| @@ -147,6 +147,17 @@ static int brb_pop(struct bop_ring_buffer *brb, struct block_op *result) · 4 unmodified lines | ||
| 147 | 147 | result->type = bop->type; |
| 148 | 148 | result->block = bop->block; |
| 149 | 149 | |
| 150 | + return 0; | |
| 151 | +} | |
| 152 | + | |
| 153 | +static int brb_pop(struct bop_ring_buffer *brb) | |
| 154 | +{ | |
| 155 | + struct block_op *bop; | |
| 156 | + | |
| 157 | + if (brb_empty(brb)) | |
| 158 | + return -ENODATA; | |
| 159 | + | |
| 160 | + bop = brb->bops + brb->begin; | |
| 150 | 161 | brb->begin = brb_next(brb, brb->begin); |
| 151 | 162 | |
| 152 | 163 | return 0; |
| @@ -211,7 +222,7 @@ static int apply_bops(struct sm_metadata *smm) · 58 unmodified lines | ||
| 211 | 222 | while (!brb_empty(&smm->uncommitted)) { |
| 212 | 223 | struct block_op bop; |
| 213 | 224 | |
| 214 | − r = brb_pop(&smm->uncommitted, &bop); | |
| 225 | + r = brb_peek(&smm->uncommitted, &bop); | |
| 215 | 226 | if (r) { |
| 216 | 227 | DMERR("bug in bop ring buffer"); |
| 217 | 228 | break; |
| @@ -220,6 +231,8 @@ static int apply_bops(struct sm_metadata *smm) · 2 unmodified lines | ||
| 220 | 231 | r = commit_bop(smm, &bop); |
| 221 | 232 | if (r) |
| 222 | 233 | break; |
| 234 | + | |
| 235 | + brb_pop(&smm->uncommitted); | |
| 223 | 236 | } |
| 224 | 237 | |
| 225 | 238 | return r; |
| @@ -683,7 +696,6 @@ static struct dm_space_map bootstrap_ops = { · 457 unmodified lines | ||
| 683 | 696 | static int sm_metadata_extend(struct dm_space_map *sm, dm_block_t extra_blocks) |
| 684 | 697 | { |
| 685 | 698 | int r, i; |
| 686 | − enum allocation_event ev; | |
| 687 | 699 | struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); |
| 688 | 700 | dm_block_t old_len = smm->ll.nr_blocks; |
| 689 | 701 | |
| @@ -705,11 +717,12 @@ static int sm_metadata_extend(struct dm_space_map *sm, dm_block_t extra_blocks) · 15 unmodified lines | ||
| 705 | 717 | * allocate any new blocks. |
| 706 | 718 | */ |
| 707 | 719 | do { |
| 708 | − for (i = old_len; !r && i < smm->begin; i++) { | |
| 709 | − r = sm_ll_inc(&smm->ll, i, &ev); | |
| 710 | − if (r) | |
| 711 | − goto out; | |
| 712 | − } | |
| 720 | + for (i = old_len; !r && i < smm->begin; i++) | |
| 721 | + r = add_bop(smm, BOP_INC, i); | |
| 722 | + | |
| 723 | + if (r) | |
| 724 | + goto out; | |
| 725 | + | |
| 713 | 726 | old_len = smm->begin; |
| 714 | 727 | |
| 715 | 728 | r = apply_bops(smm); |
| @@ -754,7 +767,6 @@ int dm_sm_metadata_create(struct dm_space_map *sm, · 38 unmodified lines | ||
| 754 | 767 | { |
| 755 | 768 | int r; |
| 756 | 769 | dm_block_t i; |
| 757 | − enum allocation_event ev; | |
| 758 | 770 | struct sm_metadata *smm = container_of(sm, struct sm_metadata, sm); |
| 759 | 771 | |
| 760 | 772 | smm->begin = superblock + 1; |
| @@ -782,7 +794,7 @@ int dm_sm_metadata_create(struct dm_space_map *sm, · 21 unmodified lines | ||
| 782 | 794 | * allocated blocks that they were built from. |
| 783 | 795 | */ |
| 784 | 796 | for (i = superblock; !r && i < smm->begin; i++) |
| 785 | − r = sm_ll_inc(&smm->ll, i, &ev); | |
| 797 | + r = add_bop(smm, BOP_INC, i); | |
| 786 | 798 | |
| 787 | 799 | if (r) |
| 788 | 800 | return r; |
| @@ -497,6 +497,7 @@ static u64 calculate_sr(struct cxl_context *ctx) · 496 unmodified lines | ||
| 497 | 497 | { |
| 498 | 498 | u64 sr = 0; |
| 499 | 499 | |
| 500 | + set_endian(sr); | |
| 500 | 501 | if (ctx->master) |
| 501 | 502 | sr |= CXL_PSL_SR_An_MP; |
| 502 | 503 | if (mfspr(SPRN_LPCR) & LPCR_TC) |
| @@ -506,7 +507,6 @@ static u64 calculate_sr(struct cxl_context *ctx) · 3 unmodified lines | ||
| 506 | 507 | sr |= CXL_PSL_SR_An_HV; |
| 507 | 508 | } else { |
| 508 | 509 | sr |= CXL_PSL_SR_An_PR | CXL_PSL_SR_An_R; |
| 509 | − set_endian(sr); | |
| 510 | 510 | sr &= ~(CXL_PSL_SR_An_HV); |
| 511 | 511 | if (!test_tsk_thread_flag(current, TIF_32BIT)) |
| 512 | 512 | sr |= CXL_PSL_SR_An_SF; |
| @@ -501,8 +501,6 @@ static int bfin_can_err(struct net_device *dev, u16 isrc, u16 status) · 500 unmodified lines | ||
| 501 | 501 | cf->data[2] |= CAN_ERR_PROT_FORM; |
| 502 | 502 | else if (status & SER) |
| 503 | 503 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 504 | − else | |
| 505 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 506 | 504 | } |
| 507 | 505 | |
| 508 | 506 | priv->can.state = state; |
| @@ -962,7 +962,6 @@ static int c_can_handle_bus_err(struct net_device *dev, · 961 unmodified lines | ||
| 962 | 962 | * type of the last error to occur on the CAN bus |
| 963 | 963 | */ |
| 964 | 964 | cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR; |
| 965 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 966 | 965 | |
| 967 | 966 | switch (lec_type) { |
| 968 | 967 | case LEC_STUFF_ERROR: |
| @@ -975,8 +974,7 @@ static int c_can_handle_bus_err(struct net_device *dev, · 6 unmodified lines | ||
| 975 | 974 | break; |
| 976 | 975 | case LEC_ACK_ERROR: |
| 977 | 976 | netdev_dbg(dev, "ack error\n"); |
| 978 | − cf->data[3] |= (CAN_ERR_PROT_LOC_ACK | | |
| 979 | − CAN_ERR_PROT_LOC_ACK_DEL); | |
| 977 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 980 | 978 | break; |
| 981 | 979 | case LEC_BIT1_ERROR: |
| 982 | 980 | netdev_dbg(dev, "bit1 error\n"); |
| @@ -988,8 +986,7 @@ static int c_can_handle_bus_err(struct net_device *dev, · 5 unmodified lines | ||
| 988 | 986 | break; |
| 989 | 987 | case LEC_CRC_ERROR: |
| 990 | 988 | netdev_dbg(dev, "CRC error\n"); |
| 991 | − cf->data[3] |= (CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 992 | − CAN_ERR_PROT_LOC_CRC_DEL); | |
| 989 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 993 | 990 | break; |
| 994 | 991 | default: |
| 995 | 992 | break; |
| @@ -578,7 +578,7 @@ static int cc770_err(struct net_device *dev, u8 status) · 577 unmodified lines | ||
| 578 | 578 | cf->data[2] |= CAN_ERR_PROT_BIT0; |
| 579 | 579 | break; |
| 580 | 580 | case STAT_LEC_CRC: |
| 581 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 581 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 582 | 582 | break; |
| 583 | 583 | } |
| 584 | 584 | } |
| @@ -535,13 +535,13 @@ static void do_bus_err(struct net_device *dev, · 534 unmodified lines | ||
| 535 | 535 | if (reg_esr & FLEXCAN_ESR_ACK_ERR) { |
| 536 | 536 | netdev_dbg(dev, "ACK_ERR irq\n"); |
| 537 | 537 | cf->can_id |= CAN_ERR_ACK; |
| 538 | − cf->data[3] |= CAN_ERR_PROT_LOC_ACK; | |
| 538 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 539 | 539 | tx_errors = 1; |
| 540 | 540 | } |
| 541 | 541 | if (reg_esr & FLEXCAN_ESR_CRC_ERR) { |
| 542 | 542 | netdev_dbg(dev, "CRC_ERR irq\n"); |
| 543 | 543 | cf->data[2] |= CAN_ERR_PROT_BIT; |
| 544 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 544 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 545 | 545 | rx_errors = 1; |
| 546 | 546 | } |
| 547 | 547 | if (reg_esr & FLEXCAN_ESR_FRM_ERR) { |
| @@ -1096,7 +1096,6 @@ static int ican3_handle_cevtind(struct ican3_dev *mod, struct ican3_msg *msg) · 1095 unmodified lines | ||
| 1096 | 1096 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 1097 | 1097 | break; |
| 1098 | 1098 | default: |
| 1099 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 1100 | 1099 | cf->data[3] = ecc & ECC_SEG; |
| 1101 | 1100 | break; |
| 1102 | 1101 | } |
| @@ -487,7 +487,6 @@ static int m_can_handle_lec_err(struct net_device *dev, · 486 unmodified lines | ||
| 487 | 487 | * type of the last error to occur on the CAN bus |
| 488 | 488 | */ |
| 489 | 489 | cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR; |
| 490 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 491 | 490 | |
| 492 | 491 | switch (lec_type) { |
| 493 | 492 | case LEC_STUFF_ERROR: |
| @@ -500,8 +499,7 @@ static int m_can_handle_lec_err(struct net_device *dev, · 6 unmodified lines | ||
| 500 | 499 | break; |
| 501 | 500 | case LEC_ACK_ERROR: |
| 502 | 501 | netdev_dbg(dev, "ack error\n"); |
| 503 | − cf->data[3] |= (CAN_ERR_PROT_LOC_ACK | | |
| 504 | − CAN_ERR_PROT_LOC_ACK_DEL); | |
| 502 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 505 | 503 | break; |
| 506 | 504 | case LEC_BIT1_ERROR: |
| 507 | 505 | netdev_dbg(dev, "bit1 error\n"); |
| @@ -513,8 +511,7 @@ static int m_can_handle_lec_err(struct net_device *dev, · 5 unmodified lines | ||
| 513 | 511 | break; |
| 514 | 512 | case LEC_CRC_ERROR: |
| 515 | 513 | netdev_dbg(dev, "CRC error\n"); |
| 516 | − cf->data[3] |= (CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 517 | − CAN_ERR_PROT_LOC_CRC_DEL); | |
| 514 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 518 | 515 | break; |
| 519 | 516 | default: |
| 520 | 517 | break; |
| @@ -559,8 +559,7 @@ static void pch_can_error(struct net_device *ndev, u32 status) · 558 unmodified lines | ||
| 559 | 559 | stats->rx_errors++; |
| 560 | 560 | break; |
| 561 | 561 | case PCH_CRC_ERR: |
| 562 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 563 | − CAN_ERR_PROT_LOC_CRC_DEL; | |
| 562 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 564 | 563 | priv->can.can_stats.bus_error++; |
| 565 | 564 | stats->rx_errors++; |
| 566 | 565 | break; |
| @@ -241,17 +241,16 @@ static void rcar_can_error(struct net_device *ndev) · 240 unmodified lines | ||
| 241 | 241 | u8 ecsr; |
| 242 | 242 | |
| 243 | 243 | netdev_dbg(priv->ndev, "Bus error interrupt:\n"); |
| 244 | − if (skb) { | |
| 244 | + if (skb) | |
| 245 | 245 | cf->can_id |= CAN_ERR_BUSERROR | CAN_ERR_PROT; |
| 246 | − cf->data[2] = CAN_ERR_PROT_UNSPEC; | |
| 247 | − } | |
| 246 | + | |
| 248 | 247 | ecsr = readb(&priv->regs->ecsr); |
| 249 | 248 | if (ecsr & RCAR_CAN_ECSR_ADEF) { |
| 250 | 249 | netdev_dbg(priv->ndev, "ACK Delimiter Error\n"); |
| 251 | 250 | tx_errors++; |
| 252 | 251 | writeb(~RCAR_CAN_ECSR_ADEF, &priv->regs->ecsr); |
| 253 | 252 | if (skb) |
| 254 | − cf->data[3] |= CAN_ERR_PROT_LOC_ACK_DEL; | |
| 253 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK_DEL; | |
| 255 | 254 | } |
| 256 | 255 | if (ecsr & RCAR_CAN_ECSR_BE0F) { |
| 257 | 256 | netdev_dbg(priv->ndev, "Bit Error (dominant)\n"); |
| @@ -272,7 +271,7 @@ static void rcar_can_error(struct net_device *ndev) · 14 unmodified lines | ||
| 272 | 271 | rx_errors++; |
| 273 | 272 | writeb(~RCAR_CAN_ECSR_CEF, &priv->regs->ecsr); |
| 274 | 273 | if (skb) |
| 275 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 274 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 276 | 275 | } |
| 277 | 276 | if (ecsr & RCAR_CAN_ECSR_AEF) { |
| 278 | 277 | netdev_dbg(priv->ndev, "ACK Error\n"); |
| @@ -280,7 +279,7 @@ static void rcar_can_error(struct net_device *ndev) · 1 unmodified line | ||
| 280 | 279 | writeb(~RCAR_CAN_ECSR_AEF, &priv->regs->ecsr); |
| 281 | 280 | if (skb) { |
| 282 | 281 | cf->can_id |= CAN_ERR_ACK; |
| 283 | − cf->data[3] |= CAN_ERR_PROT_LOC_ACK; | |
| 282 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 284 | 283 | } |
| 285 | 284 | } |
| 286 | 285 | if (ecsr & RCAR_CAN_ECSR_FEF) { |
| @@ -218,6 +218,9 @@ static void sja1000_start(struct net_device *dev) · 217 unmodified lines | ||
| 218 | 218 | priv->write_reg(priv, SJA1000_RXERR, 0x0); |
| 219 | 219 | priv->read_reg(priv, SJA1000_ECC); |
| 220 | 220 | |
| 221 | + /* clear interrupt flags */ | |
| 222 | + priv->read_reg(priv, SJA1000_IR); | |
| 223 | + | |
| 221 | 224 | /* leave reset mode */ |
| 222 | 225 | set_normal_mode(dev); |
| 223 | 226 | } |
| @@ -446,7 +449,6 @@ static int sja1000_err(struct net_device *dev, uint8_t isrc, uint8_t status) · 222 unmodified lines | ||
| 446 | 449 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 447 | 450 | break; |
| 448 | 451 | default: |
| 449 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 450 | 452 | cf->data[3] = ecc & ECC_SEG; |
| 451 | 453 | break; |
| 452 | 454 | } |
| @@ -575,7 +575,6 @@ static int sun4i_can_err(struct net_device *dev, u8 isrc, u8 status) · 574 unmodified lines | ||
| 575 | 575 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 576 | 576 | break; |
| 577 | 577 | default: |
| 578 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 579 | 578 | cf->data[3] = (ecc & SUN4I_STA_ERR_SEG_CODE) |
| 580 | 579 | >> 16; |
| 581 | 580 | break; |
| @@ -722,7 +722,6 @@ static int ti_hecc_error(struct net_device *ndev, int int_status, · 721 unmodified lines | ||
| 722 | 722 | if (err_status & HECC_BUS_ERROR) { |
| 723 | 723 | ++priv->can.can_stats.bus_error; |
| 724 | 724 | cf->can_id |= CAN_ERR_BUSERROR | CAN_ERR_PROT; |
| 725 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 726 | 725 | if (err_status & HECC_CANES_FE) { |
| 727 | 726 | hecc_set_bit(priv, HECC_CANES, HECC_CANES_FE); |
| 728 | 727 | cf->data[2] |= CAN_ERR_PROT_FORM; |
| @@ -737,13 +736,11 @@ static int ti_hecc_error(struct net_device *ndev, int int_status, · 8 unmodified lines | ||
| 737 | 736 | } |
| 738 | 737 | if (err_status & HECC_CANES_CRCE) { |
| 739 | 738 | hecc_set_bit(priv, HECC_CANES, HECC_CANES_CRCE); |
| 740 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 741 | − CAN_ERR_PROT_LOC_CRC_DEL; | |
| 739 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 742 | 740 | } |
| 743 | 741 | if (err_status & HECC_CANES_ACKE) { |
| 744 | 742 | hecc_set_bit(priv, HECC_CANES, HECC_CANES_ACKE); |
| 745 | − cf->data[3] |= CAN_ERR_PROT_LOC_ACK | | |
| 746 | − CAN_ERR_PROT_LOC_ACK_DEL; | |
| 743 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 747 | 744 | } |
| 748 | 745 | } |
| 749 | 746 | |
| @@ -377,7 +377,6 @@ static void ems_usb_rx_err(struct ems_usb *dev, struct ems_cpc_msg *msg) · 376 unmodified lines | ||
| 377 | 377 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 378 | 378 | break; |
| 379 | 379 | default: |
| 380 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 381 | 380 | cf->data[3] = ecc & SJA1000_ECC_SEG; |
| 382 | 381 | break; |
| 383 | 382 | } |
| @@ -282,7 +282,6 @@ static void esd_usb2_rx_event(struct esd_usb2_net_priv *priv, · 281 unmodified lines | ||
| 282 | 282 | cf->data[2] |= CAN_ERR_PROT_STUFF; |
| 283 | 283 | break; |
| 284 | 284 | default: |
| 285 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 286 | 285 | cf->data[3] = ecc & SJA1000_ECC_SEG; |
| 287 | 286 | break; |
| 288 | 287 | } |
| @@ -944,10 +944,9 @@ static void kvaser_usb_rx_error(const struct kvaser_usb *dev, · 943 unmodified lines | ||
| 944 | 944 | cf->can_id |= CAN_ERR_BUSERROR | CAN_ERR_PROT; |
| 945 | 945 | |
| 946 | 946 | if (es->leaf.error_factor & M16C_EF_ACKE) |
| 947 | − cf->data[3] |= (CAN_ERR_PROT_LOC_ACK); | |
| 947 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 948 | 948 | if (es->leaf.error_factor & M16C_EF_CRCE) |
| 949 | − cf->data[3] |= (CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 950 | − CAN_ERR_PROT_LOC_CRC_DEL); | |
| 949 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 951 | 950 | if (es->leaf.error_factor & M16C_EF_FORME) |
| 952 | 951 | cf->data[2] |= CAN_ERR_PROT_FORM; |
| 953 | 952 | if (es->leaf.error_factor & M16C_EF_STFE) |
| @@ -401,9 +401,7 @@ static void usb_8dev_rx_err_msg(struct usb_8dev_priv *priv, · 400 unmodified lines | ||
| 401 | 401 | tx_errors = 1; |
| 402 | 402 | break; |
| 403 | 403 | case USB_8DEV_STATUSMSG_CRC: |
| 404 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 405 | − cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 406 | − CAN_ERR_PROT_LOC_CRC_DEL; | |
| 404 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 407 | 405 | rx_errors = 1; |
| 408 | 406 | break; |
| 409 | 407 | case USB_8DEV_STATUSMSG_BIT0: |
| @@ -608,17 +608,15 @@ static void xcan_err_interrupt(struct net_device *ndev, u32 isr) · 607 unmodified lines | ||
| 608 | 608 | |
| 609 | 609 | /* Check for error interrupt */ |
| 610 | 610 | if (isr & XCAN_IXR_ERROR_MASK) { |
| 611 | − if (skb) { | |
| 611 | + if (skb) | |
| 612 | 612 | cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR; |
| 613 | − cf->data[2] |= CAN_ERR_PROT_UNSPEC; | |
| 614 | − } | |
| 615 | 613 | |
| 616 | 614 | /* Check for Ack error interrupt */ |
| 617 | 615 | if (err_status & XCAN_ESR_ACKER_MASK) { |
| 618 | 616 | stats->tx_errors++; |
| 619 | 617 | if (skb) { |
| 620 | 618 | cf->can_id |= CAN_ERR_ACK; |
| 621 | − cf->data[3] |= CAN_ERR_PROT_LOC_ACK; | |
| 619 | + cf->data[3] = CAN_ERR_PROT_LOC_ACK; | |
| 622 | 620 | } |
| 623 | 621 | } |
| 624 | 622 | |
| @@ -654,8 +652,7 @@ static void xcan_err_interrupt(struct net_device *ndev, u32 isr) · 29 unmodified lines | ||
| 654 | 652 | stats->rx_errors++; |
| 655 | 653 | if (skb) { |
| 656 | 654 | cf->can_id |= CAN_ERR_PROT; |
| 657 | − cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ | | |
| 658 | − CAN_ERR_PROT_LOC_CRC_DEL; | |
| 655 | + cf->data[3] = CAN_ERR_PROT_LOC_CRC_SEQ; | |
| 659 | 656 | } |
| 660 | 657 | } |
| 661 | 658 | priv->can.can_stats.bus_error++; |
| @@ -29,6 +29,7 @@ source "drivers/net/ethernet/apm/Kconfig" · 28 unmodified lines | ||
| 29 | 29 | source "drivers/net/ethernet/apple/Kconfig" |
| 30 | 30 | source "drivers/net/ethernet/arc/Kconfig" |
| 31 | 31 | source "drivers/net/ethernet/atheros/Kconfig" |
| 32 | +source "drivers/net/ethernet/aurora/Kconfig" | |
| 32 | 33 | source "drivers/net/ethernet/cadence/Kconfig" |
| 33 | 34 | source "drivers/net/ethernet/adi/Kconfig" |
| 34 | 35 | source "drivers/net/ethernet/broadcom/Kconfig" |
| @@ -15,6 +15,7 @@ obj-$(CONFIG_NET_XGENE) += apm/ · 14 unmodified lines | ||
| 15 | 15 | obj-$(CONFIG_NET_VENDOR_APPLE) += apple/ |
| 16 | 16 | obj-$(CONFIG_NET_VENDOR_ARC) += arc/ |
| 17 | 17 | obj-$(CONFIG_NET_VENDOR_ATHEROS) += atheros/ |
| 18 | +obj-$(CONFIG_NET_VENDOR_AURORA) += aurora/ | |
| 18 | 19 | obj-$(CONFIG_NET_CADENCE) += cadence/ |
| 19 | 20 | obj-$(CONFIG_NET_BFIN) += adi/ |
| 20 | 21 | obj-$(CONFIG_NET_VENDOR_BROADCOM) += broadcom/ |
| @@ -450,12 +450,12 @@ static netdev_tx_t xgene_enet_start_xmit(struct sk_buff *skb, · 449 unmodified lines | ||
| 450 | 450 | return NETDEV_TX_OK; |
| 451 | 451 | } |
| 452 | 452 | |
| 453 | − pdata->ring_ops->wr_cmd(tx_ring, count); | |
| 454 | 453 | skb_tx_timestamp(skb); |
| 455 | 454 | |
| 456 | 455 | pdata->stats.tx_packets++; |
| 457 | 456 | pdata->stats.tx_bytes += skb->len; |
| 458 | 457 | |
| 458 | + pdata->ring_ops->wr_cmd(tx_ring, count); | |
| 459 | 459 | return NETDEV_TX_OK; |
| 460 | 460 | } |
| 461 | 461 | |
| @@ -688,10 +688,10 @@ static int xgene_enet_open(struct net_device *ndev) · 226 unmodified lines | ||
| 688 | 688 | mac_ops->tx_enable(pdata); |
| 689 | 689 | mac_ops->rx_enable(pdata); |
| 690 | 690 | |
| 691 | + xgene_enet_napi_enable(pdata); | |
| 691 | 692 | ret = xgene_enet_register_irq(ndev); |
| 692 | 693 | if (ret) |
| 693 | 694 | return ret; |
| 694 | − xgene_enet_napi_enable(pdata); | |
| 695 | 695 | |
| 696 | 696 | if (pdata->phy_mode == PHY_INTERFACE_MODE_RGMII) |
| 697 | 697 | phy_start(pdata->phy_dev); |
| @@ -715,13 +715,13 @@ static int xgene_enet_close(struct net_device *ndev) · 17 unmodified lines | ||
| 715 | 715 | else |
| 716 | 716 | cancel_delayed_work_sync(&pdata->link_work); |
| 717 | 717 | |
| 718 | − xgene_enet_napi_disable(pdata); | |
| 719 | − xgene_enet_free_irq(ndev); | |
| 720 | − xgene_enet_process_ring(pdata->rx_ring, -1); | |
| 721 | − | |
| 722 | 718 | mac_ops->tx_disable(pdata); |
| 723 | 719 | mac_ops->rx_disable(pdata); |
| 724 | 720 | |
| 721 | + xgene_enet_free_irq(ndev); | |
| 722 | + xgene_enet_napi_disable(pdata); | |
| 723 | + xgene_enet_process_ring(pdata->rx_ring, -1); | |
| 724 | + | |
| 725 | 725 | return 0; |
| 726 | 726 | } |
| 727 | 727 | |
| @@ -1474,15 +1474,15 @@ static int xgene_enet_probe(struct platform_device *pdev) · 746 unmodified lines | ||
| 1474 | 1474 | } |
| 1475 | 1475 | ndev->hw_features = ndev->features; |
| 1476 | 1476 | |
| 1477 | − ret = register_netdev(ndev); | |
| 1477 | + ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(64)); | |
| 1478 | 1478 | if (ret) { |
| 1479 | − netdev_err(ndev, "Failed to register netdev\n"); | |
| 1479 | + netdev_err(ndev, "No usable DMA configuration\n"); | |
| 1480 | 1480 | goto err; |
| 1481 | 1481 | } |
| 1482 | 1482 | |
| 1483 | − ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(64)); | |
| 1483 | + ret = register_netdev(ndev); | |
| 1484 | 1484 | if (ret) { |
| 1485 | − netdev_err(ndev, "No usable DMA configuration\n"); | |
| 1485 | + netdev_err(ndev, "Failed to register netdev\n"); | |
| 1486 | 1486 | goto err; |
| 1487 | 1487 | } |
| 1488 | 1488 | |
| @@ -1490,14 +1490,17 @@ static int xgene_enet_probe(struct platform_device *pdev) · 1 unmodified line | ||
| 1490 | 1490 | if (ret) |
| 1491 | 1491 | goto err; |
| 1492 | 1492 | |
| 1493 | − xgene_enet_napi_add(pdata); | |
| 1494 | 1493 | mac_ops = pdata->mac_ops; |
| 1495 | − if (pdata->phy_mode == PHY_INTERFACE_MODE_RGMII) | |
| 1494 | + if (pdata->phy_mode == PHY_INTERFACE_MODE_RGMII) { | |
| 1496 | 1495 | ret = xgene_enet_mdio_config(pdata); |
| 1497 | − else | |
| 1496 | + if (ret) | |
| 1497 | + goto err; | |
| 1498 | + } else { | |
| 1498 | 1499 | INIT_DELAYED_WORK(&pdata->link_work, mac_ops->link_state); |
| 1500 | + } | |
| 1499 | 1501 | |
| 1500 | − return ret; | |
| 1502 | + xgene_enet_napi_add(pdata); | |
| 1503 | + return 0; | |
| 1501 | 1504 | err: |
| 1502 | 1505 | unregister_netdev(ndev); |
| 1503 | 1506 | free_netdev(ndev); |
| @@ -1534,6 +1534,8 @@ static const struct pci_device_id alx_pci_tbl[] = { · 1533 unmodified lines | ||
| 1534 | 1534 | .driver_data = ALX_DEV_QUIRK_MSI_INTX_DISABLE_BUG }, |
| 1535 | 1535 | { PCI_VDEVICE(ATTANSIC, ALX_DEV_ID_E2200), |
| 1536 | 1536 | .driver_data = ALX_DEV_QUIRK_MSI_INTX_DISABLE_BUG }, |
| 1537 | + { PCI_VDEVICE(ATTANSIC, ALX_DEV_ID_E2400), | |
| 1538 | + .driver_data = ALX_DEV_QUIRK_MSI_INTX_DISABLE_BUG }, | |
| 1537 | 1539 | { PCI_VDEVICE(ATTANSIC, ALX_DEV_ID_AR8162), |
| 1538 | 1540 | .driver_data = ALX_DEV_QUIRK_MSI_INTX_DISABLE_BUG }, |
| 1539 | 1541 | { PCI_VDEVICE(ATTANSIC, ALX_DEV_ID_AR8171) }, |
| @@ -37,6 +37,7 @@ · 36 unmodified lines | ||
| 37 | 37 | |
| 38 | 38 | #define ALX_DEV_ID_AR8161 0x1091 |
| 39 | 39 | #define ALX_DEV_ID_E2200 0xe091 |
| 40 | +#define ALX_DEV_ID_E2400 0xe0a1 | |
| 40 | 41 | #define ALX_DEV_ID_AR8162 0x1090 |
| 41 | 42 | #define ALX_DEV_ID_AR8171 0x10A1 |
| 42 | 43 | #define ALX_DEV_ID_AR8172 0x10A0 |
| @@ -0,0 +1,20 @@ | ||
| 1 | +config NET_VENDOR_AURORA | |
| 2 | + bool "Aurora VLSI devices" | |
| 3 | + help | |
| 4 | + If you have a network (Ethernet) device belonging to this class, | |
| 5 | + say Y. | |
| 6 | + | |
| 7 | + Note that the answer to this question doesn't directly affect the | |
| 8 | + kernel: saying N will just cause the configurator to skip all | |
| 9 | + questions about Aurora devices. If you say Y, you will be asked | |
| 10 | + for your specific device in the following questions. | |
| 11 | + | |
| 12 | +if NET_VENDOR_AURORA | |
| 13 | + | |
| 14 | +config AURORA_NB8800 | |
| 15 | + tristate "Aurora AU-NB8800 support" | |
| 16 | + select PHYLIB | |
| 17 | + help | |
| 18 | + Support for the AU-NB8800 gigabit Ethernet controller. | |
| 19 | + | |
| 20 | +endif | |
| @@ -0,0 +1 @@ | ||
| 1 | +obj-$(CONFIG_AURORA_NB8800) += nb8800.o | |
| @@ -0,0 +1,1552 @@ | ||
| 1 | +/* | |
| 2 | + * Copyright (C) 2015 Mans Rullgard <mans@mansr.com> | |
| 3 | + * | |
| 4 | + * Mostly rewritten, based on driver from Sigma Designs. Original | |
| 5 | + * copyright notice below. | |
| 6 | + * | |
| 7 | + * | |
| 8 | + * Driver for tangox SMP864x/SMP865x/SMP867x/SMP868x builtin Ethernet Mac. | |
| 9 | + * | |
| 10 | + * Copyright (C) 2005 Maxime Bizon <mbizon@freebox.fr> | |
| 11 | + * | |
| 12 | + * This program is free software; you can redistribute it and/or modify | |
| 13 | + * it under the terms of the GNU General Public License as published by | |
| 14 | + * the Free Software Foundation; either version 2 of the License, or | |
| 15 | + * (at your option) any later version. | |
| 16 | + * | |
| 17 | + * This program is distributed in the hope that it will be useful, | |
| 18 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 19 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
| 20 | + * GNU General Public License for more details. | |
| 21 | + */ | |
| 22 | + | |
| 23 | +#include <linux/module.h> | |
| 24 | +#include <linux/etherdevice.h> | |
| 25 | +#include <linux/delay.h> | |
| 26 | +#include <linux/ethtool.h> | |
| 27 | +#include <linux/interrupt.h> | |
| 28 | +#include <linux/platform_device.h> | |
| 29 | +#include <linux/of_device.h> | |
| 30 | +#include <linux/of_mdio.h> | |
| 31 | +#include <linux/of_net.h> | |
| 32 | +#include <linux/dma-mapping.h> | |
| 33 | +#include <linux/phy.h> | |
| 34 | +#include <linux/cache.h> | |
| 35 | +#include <linux/jiffies.h> | |
| 36 | +#include <linux/io.h> | |
| 37 | +#include <linux/iopoll.h> | |
| 38 | +#include <asm/barrier.h> | |
| 39 | + | |
| 40 | +#include "nb8800.h" | |
| 41 | + | |
| 42 | +static void nb8800_tx_done(struct net_device *dev); | |
| 43 | +static int nb8800_dma_stop(struct net_device *dev); | |
| 44 | + | |
| 45 | +static inline u8 nb8800_readb(struct nb8800_priv *priv, int reg) | |
| 46 | +{ | |
| 47 | + return readb_relaxed(priv->base + reg); | |
| 48 | +} | |
| 49 | + | |
| 50 | +static inline u32 nb8800_readl(struct nb8800_priv *priv, int reg) | |
| 51 | +{ | |
| 52 | + return readl_relaxed(priv->base + reg); | |
| 53 | +} | |
| 54 | + | |
| 55 | +static inline void nb8800_writeb(struct nb8800_priv *priv, int reg, u8 val) | |
| 56 | +{ | |
| 57 | + writeb_relaxed(val, priv->base + reg); | |
| 58 | +} | |
| 59 | + | |
| 60 | +static inline void nb8800_writew(struct nb8800_priv *priv, int reg, u16 val) | |
| 61 | +{ | |
| 62 | + writew_relaxed(val, priv->base + reg); | |
| 63 | +} | |
| 64 | + | |
| 65 | +static inline void nb8800_writel(struct nb8800_priv *priv, int reg, u32 val) | |
| 66 | +{ | |
| 67 | + writel_relaxed(val, priv->base + reg); | |
| 68 | +} | |
| 69 | + | |
| 70 | +static inline void nb8800_maskb(struct nb8800_priv *priv, int reg, | |
| 71 | + u32 mask, u32 val) | |
| 72 | +{ | |
| 73 | + u32 old = nb8800_readb(priv, reg); | |
| 74 | + u32 new = (old & ~mask) | (val & mask); | |
| 75 | + | |
| 76 | + if (new != old) | |
| 77 | + nb8800_writeb(priv, reg, new); | |
| 78 | +} | |
| 79 | + | |
| 80 | +static inline void nb8800_maskl(struct nb8800_priv *priv, int reg, | |
| 81 | + u32 mask, u32 val) | |
| 82 | +{ | |
| 83 | + u32 old = nb8800_readl(priv, reg); | |
| 84 | + u32 new = (old & ~mask) | (val & mask); | |
| 85 | + | |
| 86 | + if (new != old) | |
| 87 | + nb8800_writel(priv, reg, new); | |
| 88 | +} | |
| 89 | + | |
| 90 | +static inline void nb8800_modb(struct nb8800_priv *priv, int reg, u8 bits, | |
| 91 | + bool set) | |
| 92 | +{ | |
| 93 | + nb8800_maskb(priv, reg, bits, set ? bits : 0); | |
| 94 | +} | |
| 95 | + | |
| 96 | +static inline void nb8800_setb(struct nb8800_priv *priv, int reg, u8 bits) | |
| 97 | +{ | |
| 98 | + nb8800_maskb(priv, reg, bits, bits); | |
| 99 | +} | |
| 100 | + | |
| 101 | +static inline void nb8800_clearb(struct nb8800_priv *priv, int reg, u8 bits) | |
| 102 | +{ | |
| 103 | + nb8800_maskb(priv, reg, bits, 0); | |
| 104 | +} | |
| 105 | + | |
| 106 | +static inline void nb8800_modl(struct nb8800_priv *priv, int reg, u32 bits, | |
| 107 | + bool set) | |
| 108 | +{ | |
| 109 | + nb8800_maskl(priv, reg, bits, set ? bits : 0); | |
| 110 | +} | |
| 111 | + | |
| 112 | +static inline void nb8800_setl(struct nb8800_priv *priv, int reg, u32 bits) | |
| 113 | +{ | |
| 114 | + nb8800_maskl(priv, reg, bits, bits); | |
| 115 | +} | |
| 116 | + | |
| 117 | +static inline void nb8800_clearl(struct nb8800_priv *priv, int reg, u32 bits) | |
| 118 | +{ | |
| 119 | + nb8800_maskl(priv, reg, bits, 0); | |
| 120 | +} | |
| 121 | + | |
| 122 | +static int nb8800_mdio_wait(struct mii_bus *bus) | |
| 123 | +{ | |
| 124 | + struct nb8800_priv *priv = bus->priv; | |
| 125 | + u32 val; | |
| 126 | + | |
| 127 | + return readl_poll_timeout_atomic(priv->base + NB8800_MDIO_CMD, | |
| 128 | + val, !(val & MDIO_CMD_GO), 1, 1000); | |
| 129 | +} | |
| 130 | + | |
| 131 | +static int nb8800_mdio_cmd(struct mii_bus *bus, u32 cmd) | |
| 132 | +{ | |
| 133 | + struct nb8800_priv *priv = bus->priv; | |
| 134 | + int err; | |
| 135 | + | |
| 136 | + err = nb8800_mdio_wait(bus); | |
| 137 | + if (err) | |
| 138 | + return err; | |
| 139 | + | |
| 140 | + nb8800_writel(priv, NB8800_MDIO_CMD, cmd); | |
| 141 | + udelay(10); | |
| 142 | + nb8800_writel(priv, NB8800_MDIO_CMD, cmd | MDIO_CMD_GO); | |
| 143 | + | |
| 144 | + return nb8800_mdio_wait(bus); | |
| 145 | +} | |
| 146 | + | |
| 147 | +static int nb8800_mdio_read(struct mii_bus *bus, int phy_id, int reg) | |
| 148 | +{ | |
| 149 | + struct nb8800_priv *priv = bus->priv; | |
| 150 | + u32 val; | |
| 151 | + int err; | |
| 152 | + | |
| 153 | + err = nb8800_mdio_cmd(bus, MDIO_CMD_ADDR(phy_id) | MDIO_CMD_REG(reg)); | |
| 154 | + if (err) | |
| 155 | + return err; | |
| 156 | + | |
| 157 | + val = nb8800_readl(priv, NB8800_MDIO_STS); | |
| 158 | + if (val & MDIO_STS_ERR) | |
| 159 | + return 0xffff; | |
| 160 | + | |
| 161 | + return val & 0xffff; | |
| 162 | +} | |
| 163 | + | |
| 164 | +static int nb8800_mdio_write(struct mii_bus *bus, int phy_id, int reg, u16 val) | |
| 165 | +{ | |
| 166 | + u32 cmd = MDIO_CMD_ADDR(phy_id) | MDIO_CMD_REG(reg) | | |
| 167 | + MDIO_CMD_DATA(val) | MDIO_CMD_WR; | |
| 168 | + | |
| 169 | + return nb8800_mdio_cmd(bus, cmd); | |
| 170 | +} | |
| 171 | + | |
| 172 | +static void nb8800_mac_tx(struct net_device *dev, bool enable) | |
| 173 | +{ | |
| 174 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 175 | + | |
| 176 | + while (nb8800_readl(priv, NB8800_TXC_CR) & TCR_EN) | |
| 177 | + cpu_relax(); | |
| 178 | + | |
| 179 | + nb8800_modb(priv, NB8800_TX_CTL1, TX_EN, enable); | |
| 180 | +} | |
| 181 | + | |
| 182 | +static void nb8800_mac_rx(struct net_device *dev, bool enable) | |
| 183 | +{ | |
| 184 | + nb8800_modb(netdev_priv(dev), NB8800_RX_CTL, RX_EN, enable); | |
| 185 | +} | |
| 186 | + | |
| 187 | +static void nb8800_mac_af(struct net_device *dev, bool enable) | |
| 188 | +{ | |
| 189 | + nb8800_modb(netdev_priv(dev), NB8800_RX_CTL, RX_AF_EN, enable); | |
| 190 | +} | |
| 191 | + | |
| 192 | +static void nb8800_start_rx(struct net_device *dev) | |
| 193 | +{ | |
| 194 | + nb8800_setl(netdev_priv(dev), NB8800_RXC_CR, RCR_EN); | |
| 195 | +} | |
| 196 | + | |
| 197 | +static int nb8800_alloc_rx(struct net_device *dev, unsigned int i, bool napi) | |
| 198 | +{ | |
| 199 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 200 | + struct nb8800_rx_desc *rxd = &priv->rx_descs[i]; | |
| 201 | + struct nb8800_rx_buf *rxb = &priv->rx_bufs[i]; | |
| 202 | + int size = L1_CACHE_ALIGN(RX_BUF_SIZE); | |
| 203 | + dma_addr_t dma_addr; | |
| 204 | + struct page *page; | |
| 205 | + unsigned long offset; | |
| 206 | + void *data; | |
| 207 | + | |
| 208 | + data = napi ? napi_alloc_frag(size) : netdev_alloc_frag(size); | |
| 209 | + if (!data) | |
| 210 | + return -ENOMEM; | |
| 211 | + | |
| 212 | + page = virt_to_head_page(data); | |
| 213 | + offset = data - page_address(page); | |
| 214 | + | |
| 215 | + dma_addr = dma_map_page(&dev->dev, page, offset, RX_BUF_SIZE, | |
| 216 | + DMA_FROM_DEVICE); | |
| 217 | + | |
| 218 | + if (dma_mapping_error(&dev->dev, dma_addr)) { | |
| 219 | + skb_free_frag(data); | |
| 220 | + return -ENOMEM; | |
| 221 | + } | |
| 222 | + | |
| 223 | + rxb->page = page; | |
| 224 | + rxb->offset = offset; | |
| 225 | + rxd->desc.s_addr = dma_addr; | |
| 226 | + | |
| 227 | + return 0; | |
| 228 | +} | |
| 229 | + | |
| 230 | +static void nb8800_receive(struct net_device *dev, unsigned int i, | |
| 231 | + unsigned int len) | |
| 232 | +{ | |
| 233 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 234 | + struct nb8800_rx_desc *rxd = &priv->rx_descs[i]; | |
| 235 | + struct page *page = priv->rx_bufs[i].page; | |
| 236 | + int offset = priv->rx_bufs[i].offset; | |
| 237 | + void *data = page_address(page) + offset; | |
| 238 | + dma_addr_t dma = rxd->desc.s_addr; | |
| 239 | + struct sk_buff *skb; | |
| 240 | + unsigned int size; | |
| 241 | + int err; | |
| 242 | + | |
| 243 | + size = len <= RX_COPYBREAK ? len : RX_COPYHDR; | |
| 244 | + | |
| 245 | + skb = napi_alloc_skb(&priv->napi, size); | |
| 246 | + if (!skb) { | |
| 247 | + netdev_err(dev, "rx skb allocation failed\n"); | |
| 248 | + dev->stats.rx_dropped++; | |
| 249 | + return; | |
| 250 | + } | |
| 251 | + | |
| 252 | + if (len <= RX_COPYBREAK) { | |
| 253 | + dma_sync_single_for_cpu(&dev->dev, dma, len, DMA_FROM_DEVICE); | |
| 254 | + memcpy(skb_put(skb, len), data, len); | |
| 255 | + dma_sync_single_for_device(&dev->dev, dma, len, | |
| 256 | + DMA_FROM_DEVICE); | |
| 257 | + } else { | |
| 258 | + err = nb8800_alloc_rx(dev, i, true); | |
| 259 | + if (err) { | |
| 260 | + netdev_err(dev, "rx buffer allocation failed\n"); | |
| 261 | + dev->stats.rx_dropped++; | |
| 262 | + return; | |
| 263 | + } | |
| 264 | + | |
| 265 | + dma_unmap_page(&dev->dev, dma, RX_BUF_SIZE, DMA_FROM_DEVICE); | |
| 266 | + memcpy(skb_put(skb, RX_COPYHDR), data, RX_COPYHDR); | |
| 267 | + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page, | |
| 268 | + offset + RX_COPYHDR, len - RX_COPYHDR, | |
| 269 | + RX_BUF_SIZE); | |
| 270 | + } | |
| 271 | + | |
| 272 | + skb->protocol = eth_type_trans(skb, dev); | |
| 273 | + napi_gro_receive(&priv->napi, skb); | |
| 274 | +} | |
| 275 | + | |
| 276 | +static void nb8800_rx_error(struct net_device *dev, u32 report) | |
| 277 | +{ | |
| 278 | + if (report & RX_LENGTH_ERR) | |
| 279 | + dev->stats.rx_length_errors++; | |
| 280 | + | |
| 281 | + if (report & RX_FCS_ERR) | |
| 282 | + dev->stats.rx_crc_errors++; | |
| 283 | + | |
| 284 | + if (report & RX_FIFO_OVERRUN) | |
| 285 | + dev->stats.rx_fifo_errors++; | |
| 286 | + | |
| 287 | + if (report & RX_ALIGNMENT_ERROR) | |
| 288 | + dev->stats.rx_frame_errors++; | |
| 289 | + | |
| 290 | + dev->stats.rx_errors++; | |
| 291 | +} | |
| 292 | + | |
| 293 | +static int nb8800_poll(struct napi_struct *napi, int budget) | |
| 294 | +{ | |
| 295 | + struct net_device *dev = napi->dev; | |
| 296 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 297 | + struct nb8800_rx_desc *rxd; | |
| 298 | + unsigned int last = priv->rx_eoc; | |
| 299 | + unsigned int next; | |
| 300 | + int work = 0; | |
| 301 | + | |
| 302 | + nb8800_tx_done(dev); | |
| 303 | + | |
| 304 | +again: | |
| 305 | + while (work < budget) { | |
| 306 | + struct nb8800_rx_buf *rxb; | |
| 307 | + unsigned int len; | |
| 308 | + | |
| 309 | + next = (last + 1) % RX_DESC_COUNT; | |
| 310 | + | |
| 311 | + rxb = &priv->rx_bufs[next]; | |
| 312 | + rxd = &priv->rx_descs[next]; | |
| 313 | + | |
| 314 | + if (!rxd->report) | |
| 315 | + break; | |
| 316 | + | |
| 317 | + len = RX_BYTES_TRANSFERRED(rxd->report); | |
| 318 | + | |
| 319 | + if (IS_RX_ERROR(rxd->report)) | |
| 320 | + nb8800_rx_error(dev, rxd->report); | |
| 321 | + else | |
| 322 | + nb8800_receive(dev, next, len); | |
| 323 | + | |
| 324 | + dev->stats.rx_packets++; | |
| 325 | + dev->stats.rx_bytes += len; | |
| 326 | + | |
| 327 | + if (rxd->report & RX_MULTICAST_PKT) | |
| 328 | + dev->stats.multicast++; | |
| 329 | + | |
| 330 | + rxd->report = 0; | |
| 331 | + last = next; | |
| 332 | + work++; | |
| 333 | + } | |
| 334 | + | |
| 335 | + if (work) { | |
| 336 | + priv->rx_descs[last].desc.config |= DESC_EOC; | |
| 337 | + wmb(); /* ensure new EOC is written before clearing old */ | |
| 338 | + priv->rx_descs[priv->rx_eoc].desc.config &= ~DESC_EOC; | |
| 339 | + priv->rx_eoc = last; | |
| 340 | + nb8800_start_rx(dev); | |
| 341 | + } | |
| 342 | + | |
| 343 | + if (work < budget) { | |
| 344 | + nb8800_writel(priv, NB8800_RX_ITR, priv->rx_itr_irq); | |
| 345 | + | |
| 346 | + /* If a packet arrived after we last checked but | |
| 347 | + * before writing RX_ITR, the interrupt will be | |
| 348 | + * delayed, so we retrieve it now. | |
| 349 | + */ | |
| 350 | + if (priv->rx_descs[next].report) | |
| 351 | + goto again; | |
| 352 | + | |
| 353 | + napi_complete_done(napi, work); | |
| 354 | + } | |
| 355 | + | |
| 356 | + return work; | |
| 357 | +} | |
| 358 | + | |
| 359 | +static void __nb8800_tx_dma_start(struct net_device *dev) | |
| 360 | +{ | |
| 361 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 362 | + struct nb8800_tx_buf *txb; | |
| 363 | + u32 txc_cr; | |
| 364 | + | |
| 365 | + txb = &priv->tx_bufs[priv->tx_queue]; | |
| 366 | + if (!txb->ready) | |
| 367 | + return; | |
| 368 | + | |
| 369 | + txc_cr = nb8800_readl(priv, NB8800_TXC_CR); | |
| 370 | + if (txc_cr & TCR_EN) | |
| 371 | + return; | |
| 372 | + | |
| 373 | + nb8800_writel(priv, NB8800_TX_DESC_ADDR, txb->dma_desc); | |
| 374 | + wmb(); /* ensure desc addr is written before starting DMA */ | |
| 375 | + nb8800_writel(priv, NB8800_TXC_CR, txc_cr | TCR_EN); | |
| 376 | + | |
| 377 | + priv->tx_queue = (priv->tx_queue + txb->chain_len) % TX_DESC_COUNT; | |
| 378 | +} | |
| 379 | + | |
| 380 | +static void nb8800_tx_dma_start(struct net_device *dev) | |
| 381 | +{ | |
| 382 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 383 | + | |
| 384 | + spin_lock_irq(&priv->tx_lock); | |
| 385 | + __nb8800_tx_dma_start(dev); | |
| 386 | + spin_unlock_irq(&priv->tx_lock); | |
| 387 | +} | |
| 388 | + | |
| 389 | +static void nb8800_tx_dma_start_irq(struct net_device *dev) | |
| 390 | +{ | |
| 391 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 392 | + | |
| 393 | + spin_lock(&priv->tx_lock); | |
| 394 | + __nb8800_tx_dma_start(dev); | |
| 395 | + spin_unlock(&priv->tx_lock); | |
| 396 | +} | |
| 397 | + | |
| 398 | +static int nb8800_xmit(struct sk_buff *skb, struct net_device *dev) | |
| 399 | +{ | |
| 400 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 401 | + struct nb8800_tx_desc *txd; | |
| 402 | + struct nb8800_tx_buf *txb; | |
| 403 | + struct nb8800_dma_desc *desc; | |
| 404 | + dma_addr_t dma_addr; | |
| 405 | + unsigned int dma_len; | |
| 406 | + unsigned int align; | |
| 407 | + unsigned int next; | |
| 408 | + | |
| 409 | + if (atomic_read(&priv->tx_free) <= NB8800_DESC_LOW) { | |
| 410 | + netif_stop_queue(dev); | |
| 411 | + return NETDEV_TX_BUSY; | |
| 412 | + } | |
| 413 | + | |
| 414 | + align = (8 - (uintptr_t)skb->data) & 7; | |
| 415 | + | |
| 416 | + dma_len = skb->len - align; | |
| 417 | + dma_addr = dma_map_single(&dev->dev, skb->data + align, | |
| 418 | + dma_len, DMA_TO_DEVICE); | |
| 419 | + | |
| 420 | + if (dma_mapping_error(&dev->dev, dma_addr)) { | |
| 421 | + netdev_err(dev, "tx dma mapping error\n"); | |
| 422 | + kfree_skb(skb); | |
| 423 | + dev->stats.tx_dropped++; | |
| 424 | + return NETDEV_TX_OK; | |
| 425 | + } | |
| 426 | + | |
| 427 | + if (atomic_dec_return(&priv->tx_free) <= NB8800_DESC_LOW) { | |
| 428 | + netif_stop_queue(dev); | |
| 429 | + skb->xmit_more = 0; | |
| 430 | + } | |
| 431 | + | |
| 432 | + next = priv->tx_next; | |
| 433 | + txb = &priv->tx_bufs[next]; | |
| 434 | + txd = &priv->tx_descs[next]; | |
| 435 | + desc = &txd->desc[0]; | |
| 436 | + | |
| 437 | + next = (next + 1) % TX_DESC_COUNT; | |
| 438 | + | |
| 439 | + if (align) { | |
| 440 | + memcpy(txd->buf, skb->data, align); | |
| 441 | + | |
| 442 | + desc->s_addr = | |
| 443 | + txb->dma_desc + offsetof(struct nb8800_tx_desc, buf); | |
| 444 | + desc->n_addr = txb->dma_desc + sizeof(txd->desc[0]); | |
| 445 | + desc->config = DESC_BTS(2) | DESC_DS | align; | |
| 446 | + | |
| 447 | + desc++; | |
| 448 | + } | |
| 449 | + | |
| 450 | + desc->s_addr = dma_addr; | |
| 451 | + desc->n_addr = priv->tx_bufs[next].dma_desc; | |
| 452 | + desc->config = DESC_BTS(2) | DESC_DS | DESC_EOF | dma_len; | |
| 453 | + | |
| 454 | + if (!skb->xmit_more) | |
| 455 | + desc->config |= DESC_EOC; | |
| 456 | + | |
| 457 | + txb->skb = skb; | |
| 458 | + txb->dma_addr = dma_addr; | |
| 459 | + txb->dma_len = dma_len; | |
| 460 | + | |
| 461 | + if (!priv->tx_chain) { | |
| 462 | + txb->chain_len = 1; | |
| 463 | + priv->tx_chain = txb; | |
| 464 | + } else { | |
| 465 | + priv->tx_chain->chain_len++; | |
| 466 | + } | |
| 467 | + | |
| 468 | + netdev_sent_queue(dev, skb->len); | |
| 469 | + | |
| 470 | + priv->tx_next = next; | |
| 471 | + | |
| 472 | + if (!skb->xmit_more) { | |
| 473 | + smp_wmb(); | |
| 474 | + priv->tx_chain->ready = true; | |
| 475 | + priv->tx_chain = NULL; | |
| 476 | + nb8800_tx_dma_start(dev); | |
| 477 | + } | |
| 478 | + | |
| 479 | + return NETDEV_TX_OK; | |
| 480 | +} | |
| 481 | + | |
| 482 | +static void nb8800_tx_error(struct net_device *dev, u32 report) | |
| 483 | +{ | |
| 484 | + if (report & TX_LATE_COLLISION) | |
| 485 | + dev->stats.collisions++; | |
| 486 | + | |
| 487 | + if (report & TX_PACKET_DROPPED) | |
| 488 | + dev->stats.tx_dropped++; | |
| 489 | + | |
| 490 | + if (report & TX_FIFO_UNDERRUN) | |
| 491 | + dev->stats.tx_fifo_errors++; | |
| 492 | + | |
| 493 | + dev->stats.tx_errors++; | |
| 494 | +} | |
| 495 | + | |
| 496 | +static void nb8800_tx_done(struct net_device *dev) | |
| 497 | +{ | |
| 498 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 499 | + unsigned int limit = priv->tx_next; | |
| 500 | + unsigned int done = priv->tx_done; | |
| 501 | + unsigned int packets = 0; | |
| 502 | + unsigned int len = 0; | |
| 503 | + | |
| 504 | + while (done != limit) { | |
| 505 | + struct nb8800_tx_desc *txd = &priv->tx_descs[done]; | |
| 506 | + struct nb8800_tx_buf *txb = &priv->tx_bufs[done]; | |
| 507 | + struct sk_buff *skb; | |
| 508 | + | |
| 509 | + if (!txd->report) | |
| 510 | + break; | |
| 511 | + | |
| 512 | + skb = txb->skb; | |
| 513 | + len += skb->len; | |
| 514 | + | |
| 515 | + dma_unmap_single(&dev->dev, txb->dma_addr, txb->dma_len, | |
| 516 | + DMA_TO_DEVICE); | |
| 517 | + | |
| 518 | + if (IS_TX_ERROR(txd->report)) { | |
| 519 | + nb8800_tx_error(dev, txd->report); | |
| 520 | + kfree_skb(skb); | |
| 521 | + } else { | |
| 522 | + consume_skb(skb); | |
| 523 | + } | |
| 524 | + | |
| 525 | + dev->stats.tx_packets++; | |
| 526 | + dev->stats.tx_bytes += TX_BYTES_TRANSFERRED(txd->report); | |
| 527 | + dev->stats.collisions += TX_EARLY_COLLISIONS(txd->report); | |
| 528 | + | |
| 529 | + txb->skb = NULL; | |
| 530 | + txb->ready = false; | |
| 531 | + txd->report = 0; | |
| 532 | + | |
| 533 | + done = (done + 1) % TX_DESC_COUNT; | |
| 534 | + packets++; | |
| 535 | + } | |
| 536 | + | |
| 537 | + if (packets) { | |
| 538 | + smp_mb__before_atomic(); | |
| 539 | + atomic_add(packets, &priv->tx_free); | |
| 540 | + netdev_completed_queue(dev, packets, len); | |
| 541 | + netif_wake_queue(dev); | |
| 542 | + priv->tx_done = done; | |
| 543 | + } | |
| 544 | +} | |
| 545 | + | |
| 546 | +static irqreturn_t nb8800_irq(int irq, void *dev_id) | |
| 547 | +{ | |
| 548 | + struct net_device *dev = dev_id; | |
| 549 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 550 | + irqreturn_t ret = IRQ_NONE; | |
| 551 | + u32 val; | |
| 552 | + | |
| 553 | + /* tx interrupt */ | |
| 554 | + val = nb8800_readl(priv, NB8800_TXC_SR); | |
| 555 | + if (val) { | |
| 556 | + nb8800_writel(priv, NB8800_TXC_SR, val); | |
| 557 | + | |
| 558 | + if (val & TSR_DI) | |
| 559 | + nb8800_tx_dma_start_irq(dev); | |
| 560 | + | |
| 561 | + if (val & TSR_TI) | |
| 562 | + napi_schedule_irqoff(&priv->napi); | |
| 563 | + | |
| 564 | + if (unlikely(val & TSR_DE)) | |
| 565 | + netdev_err(dev, "TX DMA error\n"); | |
| 566 | + | |
| 567 | + /* should never happen with automatic status retrieval */ | |
| 568 | + if (unlikely(val & TSR_TO)) | |
| 569 | + netdev_err(dev, "TX Status FIFO overflow\n"); | |
| 570 | + | |
| 571 | + ret = IRQ_HANDLED; | |
| 572 | + } | |
| 573 | + | |
| 574 | + /* rx interrupt */ | |
| 575 | + val = nb8800_readl(priv, NB8800_RXC_SR); | |
| 576 | + if (val) { | |
| 577 | + nb8800_writel(priv, NB8800_RXC_SR, val); | |
| 578 | + | |
| 579 | + if (likely(val & (RSR_RI | RSR_DI))) { | |
| 580 | + nb8800_writel(priv, NB8800_RX_ITR, priv->rx_itr_poll); | |
| 581 | + napi_schedule_irqoff(&priv->napi); | |
| 582 | + } | |
| 583 | + | |
| 584 | + if (unlikely(val & RSR_DE)) | |
| 585 | + netdev_err(dev, "RX DMA error\n"); | |
| 586 | + | |
| 587 | + /* should never happen with automatic status retrieval */ | |
| 588 | + if (unlikely(val & RSR_RO)) | |
| 589 | + netdev_err(dev, "RX Status FIFO overflow\n"); | |
| 590 | + | |
| 591 | + ret = IRQ_HANDLED; | |
| 592 | + } | |
| 593 | + | |
| 594 | + return ret; | |
| 595 | +} | |
| 596 | + | |
| 597 | +static void nb8800_mac_config(struct net_device *dev) | |
| 598 | +{ | |
| 599 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 600 | + bool gigabit = priv->speed == SPEED_1000; | |
| 601 | + u32 mac_mode_mask = RGMII_MODE | HALF_DUPLEX | GMAC_MODE; | |
| 602 | + u32 mac_mode = 0; | |
| 603 | + u32 slot_time; | |
| 604 | + u32 phy_clk; | |
| 605 | + u32 ict; | |
| 606 | + | |
| 607 | + if (!priv->duplex) | |
| 608 | + mac_mode |= HALF_DUPLEX; | |
| 609 | + | |
| 610 | + if (gigabit) { | |
| 611 | + if (priv->phy_mode == PHY_INTERFACE_MODE_RGMII) | |
| 612 | + mac_mode |= RGMII_MODE; | |
| 613 | + | |
| 614 | + mac_mode |= GMAC_MODE; | |
| 615 | + phy_clk = 125000000; | |
| 616 | + | |
| 617 | + /* Should be 512 but register is only 8 bits */ | |
| 618 | + slot_time = 255; | |
| 619 | + } else { | |
| 620 | + phy_clk = 25000000; | |
| 621 | + slot_time = 128; | |
| 622 | + } | |
| 623 | + | |
| 624 | + ict = DIV_ROUND_UP(phy_clk, clk_get_rate(priv->clk)); | |
| 625 | + | |
| 626 | + nb8800_writeb(priv, NB8800_IC_THRESHOLD, ict); | |
| 627 | + nb8800_writeb(priv, NB8800_SLOT_TIME, slot_time); | |
| 628 | + nb8800_maskb(priv, NB8800_MAC_MODE, mac_mode_mask, mac_mode); | |
| 629 | +} | |
| 630 | + | |
| 631 | +static void nb8800_pause_config(struct net_device *dev) | |
| 632 | +{ | |
| 633 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 634 | + struct phy_device *phydev = priv->phydev; | |
| 635 | + u32 rxcr; | |
| 636 | + | |
| 637 | + if (priv->pause_aneg) { | |
| 638 | + if (!phydev || !phydev->link) | |
| 639 | + return; | |
| 640 | + | |
| 641 | + priv->pause_rx = phydev->pause; | |
| 642 | + priv->pause_tx = phydev->pause ^ phydev->asym_pause; | |
| 643 | + } | |
| 644 | + | |
| 645 | + nb8800_modb(priv, NB8800_RX_CTL, RX_PAUSE_EN, priv->pause_rx); | |
| 646 | + | |
| 647 | + rxcr = nb8800_readl(priv, NB8800_RXC_CR); | |
| 648 | + if (!!(rxcr & RCR_FL) == priv->pause_tx) | |
| 649 | + return; | |
| 650 | + | |
| 651 | + if (netif_running(dev)) { | |
| 652 | + napi_disable(&priv->napi); | |
| 653 | + netif_tx_lock_bh(dev); | |
| 654 | + nb8800_dma_stop(dev); | |
| 655 | + nb8800_modl(priv, NB8800_RXC_CR, RCR_FL, priv->pause_tx); | |
| 656 | + nb8800_start_rx(dev); | |
| 657 | + netif_tx_unlock_bh(dev); | |
| 658 | + napi_enable(&priv->napi); | |
| 659 | + } else { | |
| 660 | + nb8800_modl(priv, NB8800_RXC_CR, RCR_FL, priv->pause_tx); | |
| 661 | + } | |
| 662 | +} | |
| 663 | + | |
| 664 | +static void nb8800_link_reconfigure(struct net_device *dev) | |
| 665 | +{ | |
| 666 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 667 | + struct phy_device *phydev = priv->phydev; | |
| 668 | + int change = 0; | |
| 669 | + | |
| 670 | + if (phydev->link) { | |
| 671 | + if (phydev->speed != priv->speed) { | |
| 672 | + priv->speed = phydev->speed; | |
| 673 | + change = 1; | |
| 674 | + } | |
| 675 | + | |
| 676 | + if (phydev->duplex != priv->duplex) { | |
| 677 | + priv->duplex = phydev->duplex; | |
| 678 | + change = 1; | |
| 679 | + } | |
| 680 | + | |
| 681 | + if (change) | |
| 682 | + nb8800_mac_config(dev); | |
| 683 | + | |
| 684 | + nb8800_pause_config(dev); | |
| 685 | + } | |
| 686 | + | |
| 687 | + if (phydev->link != priv->link) { | |
| 688 | + priv->link = phydev->link; | |
| 689 | + change = 1; | |
| 690 | + } | |
| 691 | + | |
| 692 | + if (change) | |
| 693 | + phy_print_status(priv->phydev); | |
| 694 | +} | |
| 695 | + | |
| 696 | +static void nb8800_update_mac_addr(struct net_device *dev) | |
| 697 | +{ | |
| 698 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 699 | + int i; | |
| 700 | + | |
| 701 | + for (i = 0; i < ETH_ALEN; i++) | |
| 702 | + nb8800_writeb(priv, NB8800_SRC_ADDR(i), dev->dev_addr[i]); | |
| 703 | + | |
| 704 | + for (i = 0; i < ETH_ALEN; i++) | |
| 705 | + nb8800_writeb(priv, NB8800_UC_ADDR(i), dev->dev_addr[i]); | |
| 706 | +} | |
| 707 | + | |
| 708 | +static int nb8800_set_mac_address(struct net_device *dev, void *addr) | |
| 709 | +{ | |
| 710 | + struct sockaddr *sock = addr; | |
| 711 | + | |
| 712 | + if (netif_running(dev)) | |
| 713 | + return -EBUSY; | |
| 714 | + | |
| 715 | + ether_addr_copy(dev->dev_addr, sock->sa_data); | |
| 716 | + nb8800_update_mac_addr(dev); | |
| 717 | + | |
| 718 | + return 0; | |
| 719 | +} | |
| 720 | + | |
| 721 | +static void nb8800_mc_init(struct net_device *dev, int val) | |
| 722 | +{ | |
| 723 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 724 | + | |
| 725 | + nb8800_writeb(priv, NB8800_MC_INIT, val); | |
| 726 | + readb_poll_timeout_atomic(priv->base + NB8800_MC_INIT, val, !val, | |
| 727 | + 1, 1000); | |
| 728 | +} | |
| 729 | + | |
| 730 | +static void nb8800_set_rx_mode(struct net_device *dev) | |
| 731 | +{ | |
| 732 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 733 | + struct netdev_hw_addr *ha; | |
| 734 | + int i; | |
| 735 | + | |
| 736 | + if (dev->flags & (IFF_PROMISC | IFF_ALLMULTI)) { | |
| 737 | + nb8800_mac_af(dev, false); | |
| 738 | + return; | |
| 739 | + } | |
| 740 | + | |
| 741 | + nb8800_mac_af(dev, true); | |
| 742 | + nb8800_mc_init(dev, 0); | |
| 743 | + | |
| 744 | + netdev_for_each_mc_addr(ha, dev) { | |
| 745 | + for (i = 0; i < ETH_ALEN; i++) | |
| 746 | + nb8800_writeb(priv, NB8800_MC_ADDR(i), ha->addr[i]); | |
| 747 | + | |
| 748 | + nb8800_mc_init(dev, 0xff); | |
| 749 | + } | |
| 750 | +} | |
| 751 | + | |
| 752 | +#define RX_DESC_SIZE (RX_DESC_COUNT * sizeof(struct nb8800_rx_desc)) | |
| 753 | +#define TX_DESC_SIZE (TX_DESC_COUNT * sizeof(struct nb8800_tx_desc)) | |
| 754 | + | |
| 755 | +static void nb8800_dma_free(struct net_device *dev) | |
| 756 | +{ | |
| 757 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 758 | + unsigned int i; | |
| 759 | + | |
| 760 | + if (priv->rx_bufs) { | |
| 761 | + for (i = 0; i < RX_DESC_COUNT; i++) | |
| 762 | + if (priv->rx_bufs[i].page) | |
| 763 | + put_page(priv->rx_bufs[i].page); | |
| 764 | + | |
| 765 | + kfree(priv->rx_bufs); | |
| 766 | + priv->rx_bufs = NULL; | |
| 767 | + } | |
| 768 | + | |
| 769 | + if (priv->tx_bufs) { | |
| 770 | + for (i = 0; i < TX_DESC_COUNT; i++) | |
| 771 | + kfree_skb(priv->tx_bufs[i].skb); | |
| 772 | + | |
| 773 | + kfree(priv->tx_bufs); | |
| 774 | + priv->tx_bufs = NULL; | |
| 775 | + } | |
| 776 | + | |
| 777 | + if (priv->rx_descs) { | |
| 778 | + dma_free_coherent(dev->dev.parent, RX_DESC_SIZE, priv->rx_descs, | |
| 779 | + priv->rx_desc_dma); | |
| 780 | + priv->rx_descs = NULL; | |
| 781 | + } | |
| 782 | + | |
| 783 | + if (priv->tx_descs) { | |
| 784 | + dma_free_coherent(dev->dev.parent, TX_DESC_SIZE, priv->tx_descs, | |
| 785 | + priv->tx_desc_dma); | |
| 786 | + priv->tx_descs = NULL; | |
| 787 | + } | |
| 788 | +} | |
| 789 | + | |
| 790 | +static void nb8800_dma_reset(struct net_device *dev) | |
| 791 | +{ | |
| 792 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 793 | + struct nb8800_rx_desc *rxd; | |
| 794 | + struct nb8800_tx_desc *txd; | |
| 795 | + unsigned int i; | |
| 796 | + | |
| 797 | + for (i = 0; i < RX_DESC_COUNT; i++) { | |
| 798 | + dma_addr_t rx_dma = priv->rx_desc_dma + i * sizeof(*rxd); | |
| 799 | + | |
| 800 | + rxd = &priv->rx_descs[i]; | |
| 801 | + rxd->desc.n_addr = rx_dma + sizeof(*rxd); | |
| 802 | + rxd->desc.r_addr = | |
| 803 | + rx_dma + offsetof(struct nb8800_rx_desc, report); | |
| 804 | + rxd->desc.config = priv->rx_dma_config; | |
| 805 | + rxd->report = 0; | |
| 806 | + } | |
| 807 | + | |
| 808 | + rxd->desc.n_addr = priv->rx_desc_dma; | |
| 809 | + rxd->desc.config |= DESC_EOC; | |
| 810 | + | |
| 811 | + priv->rx_eoc = RX_DESC_COUNT - 1; | |
| 812 | + | |
| 813 | + for (i = 0; i < TX_DESC_COUNT; i++) { | |
| 814 | + struct nb8800_tx_buf *txb = &priv->tx_bufs[i]; | |
| 815 | + dma_addr_t r_dma = txb->dma_desc + | |
| 816 | + offsetof(struct nb8800_tx_desc, report); | |
| 817 | + | |
| 818 | + txd = &priv->tx_descs[i]; | |
| 819 | + txd->desc[0].r_addr = r_dma; | |
| 820 | + txd->desc[1].r_addr = r_dma; | |
| 821 | + txd->report = 0; | |
| 822 | + } | |
| 823 | + | |
| 824 | + priv->tx_next = 0; | |
| 825 | + priv->tx_queue = 0; | |
| 826 | + priv->tx_done = 0; | |
| 827 | + atomic_set(&priv->tx_free, TX_DESC_COUNT); | |
| 828 | + | |
| 829 | + nb8800_writel(priv, NB8800_RX_DESC_ADDR, priv->rx_desc_dma); | |
| 830 | + | |
| 831 | + wmb(); /* ensure all setup is written before starting */ | |
| 832 | +} | |
| 833 | + | |
| 834 | +static int nb8800_dma_init(struct net_device *dev) | |
| 835 | +{ | |
| 836 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 837 | + unsigned int n_rx = RX_DESC_COUNT; | |
| 838 | + unsigned int n_tx = TX_DESC_COUNT; | |
| 839 | + unsigned int i; | |
| 840 | + int err; | |
| 841 | + | |
| 842 | + priv->rx_descs = dma_alloc_coherent(dev->dev.parent, RX_DESC_SIZE, | |
| 843 | + &priv->rx_desc_dma, GFP_KERNEL); | |
| 844 | + if (!priv->rx_descs) | |
| 845 | + goto err_out; | |
| 846 | + | |
| 847 | + priv->rx_bufs = kcalloc(n_rx, sizeof(*priv->rx_bufs), GFP_KERNEL); | |
| 848 | + if (!priv->rx_bufs) | |
| 849 | + goto err_out; | |
| 850 | + | |
| 851 | + for (i = 0; i < n_rx; i++) { | |
| 852 | + err = nb8800_alloc_rx(dev, i, false); | |
| 853 | + if (err) | |
| 854 | + goto err_out; | |
| 855 | + } | |
| 856 | + | |
| 857 | + priv->tx_descs = dma_alloc_coherent(dev->dev.parent, TX_DESC_SIZE, | |
| 858 | + &priv->tx_desc_dma, GFP_KERNEL); | |
| 859 | + if (!priv->tx_descs) | |
| 860 | + goto err_out; | |
| 861 | + | |
| 862 | + priv->tx_bufs = kcalloc(n_tx, sizeof(*priv->tx_bufs), GFP_KERNEL); | |
| 863 | + if (!priv->tx_bufs) | |
| 864 | + goto err_out; | |
| 865 | + | |
| 866 | + for (i = 0; i < n_tx; i++) | |
| 867 | + priv->tx_bufs[i].dma_desc = | |
| 868 | + priv->tx_desc_dma + i * sizeof(struct nb8800_tx_desc); | |
| 869 | + | |
| 870 | + nb8800_dma_reset(dev); | |
| 871 | + | |
| 872 | + return 0; | |
| 873 | + | |
| 874 | +err_out: | |
| 875 | + nb8800_dma_free(dev); | |
| 876 | + | |
| 877 | + return -ENOMEM; | |
| 878 | +} | |
| 879 | + | |
| 880 | +static int nb8800_dma_stop(struct net_device *dev) | |
| 881 | +{ | |
| 882 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 883 | + struct nb8800_tx_buf *txb = &priv->tx_bufs[0]; | |
| 884 | + struct nb8800_tx_desc *txd = &priv->tx_descs[0]; | |
| 885 | + int retry = 5; | |
| 886 | + u32 txcr; | |
| 887 | + u32 rxcr; | |
| 888 | + int err; | |
| 889 | + unsigned int i; | |
| 890 | + | |
| 891 | + /* wait for tx to finish */ | |
| 892 | + err = readl_poll_timeout_atomic(priv->base + NB8800_TXC_CR, txcr, | |
| 893 | + !(txcr & TCR_EN) && | |
| 894 | + priv->tx_done == priv->tx_next, | |
| 895 | + 1000, 1000000); | |
| 896 | + if (err) | |
| 897 | + return err; | |
| 898 | + | |
| 899 | + /* The rx DMA only stops if it reaches the end of chain. | |
| 900 | + * To make this happen, we set the EOC flag on all rx | |
| 901 | + * descriptors, put the device in loopback mode, and send | |
| 902 | + * a few dummy frames. The interrupt handler will ignore | |
| 903 | + * these since NAPI is disabled and no real frames are in | |
| 904 | + * the tx queue. | |
| 905 | + */ | |
| 906 | + | |
| 907 | + for (i = 0; i < RX_DESC_COUNT; i++) | |
| 908 | + priv->rx_descs[i].desc.config |= DESC_EOC; | |
| 909 | + | |
| 910 | + txd->desc[0].s_addr = | |
| 911 | + txb->dma_desc + offsetof(struct nb8800_tx_desc, buf); | |
| 912 | + txd->desc[0].config = DESC_BTS(2) | DESC_DS | DESC_EOF | DESC_EOC | 8; | |
| 913 | + memset(txd->buf, 0, sizeof(txd->buf)); | |
| 914 | + | |
| 915 | + nb8800_mac_af(dev, false); | |
| 916 | + nb8800_setb(priv, NB8800_MAC_MODE, LOOPBACK_EN); | |
| 917 | + | |
| 918 | + do { | |
| 919 | + nb8800_writel(priv, NB8800_TX_DESC_ADDR, txb->dma_desc); | |
| 920 | + wmb(); | |
| 921 | + nb8800_writel(priv, NB8800_TXC_CR, txcr | TCR_EN); | |
| 922 | + | |
| 923 | + err = readl_poll_timeout_atomic(priv->base + NB8800_RXC_CR, | |
| 924 | + rxcr, !(rxcr & RCR_EN), | |
| 925 | + 1000, 100000); | |
| 926 | + } while (err && --retry); | |
| 927 | + | |
| 928 | + nb8800_mac_af(dev, true); | |
| 929 | + nb8800_clearb(priv, NB8800_MAC_MODE, LOOPBACK_EN); | |
| 930 | + nb8800_dma_reset(dev); | |
| 931 | + | |
| 932 | + return retry ? 0 : -ETIMEDOUT; | |
| 933 | +} | |
| 934 | + | |
| 935 | +static void nb8800_pause_adv(struct net_device *dev) | |
| 936 | +{ | |
| 937 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 938 | + u32 adv = 0; | |
| 939 | + | |
| 940 | + if (!priv->phydev) | |
| 941 | + return; | |
| 942 | + | |
| 943 | + if (priv->pause_rx) | |
| 944 | + adv |= ADVERTISED_Pause | ADVERTISED_Asym_Pause; | |
| 945 | + if (priv->pause_tx) | |
| 946 | + adv ^= ADVERTISED_Asym_Pause; | |
| 947 | + | |
| 948 | + priv->phydev->supported |= adv; | |
| 949 | + priv->phydev->advertising |= adv; | |
| 950 | +} | |
| 951 | + | |
| 952 | +static int nb8800_open(struct net_device *dev) | |
| 953 | +{ | |
| 954 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 955 | + int err; | |
| 956 | + | |
| 957 | + /* clear any pending interrupts */ | |
| 958 | + nb8800_writel(priv, NB8800_RXC_SR, 0xf); | |
| 959 | + nb8800_writel(priv, NB8800_TXC_SR, 0xf); | |
| 960 | + | |
| 961 | + err = nb8800_dma_init(dev); | |
| 962 | + if (err) | |
| 963 | + return err; | |
| 964 | + | |
| 965 | + err = request_irq(dev->irq, nb8800_irq, 0, dev_name(&dev->dev), dev); | |
| 966 | + if (err) | |
| 967 | + goto err_free_dma; | |
| 968 | + | |
| 969 | + nb8800_mac_rx(dev, true); | |
| 970 | + nb8800_mac_tx(dev, true); | |
| 971 | + | |
| 972 | + priv->phydev = of_phy_connect(dev, priv->phy_node, | |
| 973 | + nb8800_link_reconfigure, 0, | |
| 974 | + priv->phy_mode); | |
| 975 | + if (!priv->phydev) | |
| 976 | + goto err_free_irq; | |
| 977 | + | |
| 978 | + nb8800_pause_adv(dev); | |
| 979 | + | |
| 980 | + netdev_reset_queue(dev); | |
| 981 | + napi_enable(&priv->napi); | |
| 982 | + netif_start_queue(dev); | |
| 983 | + | |
| 984 | + nb8800_start_rx(dev); | |
| 985 | + phy_start(priv->phydev); | |
| 986 | + | |
| 987 | + return 0; | |
| 988 | + | |
| 989 | +err_free_irq: | |
| 990 | + free_irq(dev->irq, dev); | |
| 991 | +err_free_dma: | |
| 992 | + nb8800_dma_free(dev); | |
| 993 | + | |
| 994 | + return err; | |
| 995 | +} | |
| 996 | + | |
| 997 | +static int nb8800_stop(struct net_device *dev) | |
| 998 | +{ | |
| 999 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1000 | + | |
| 1001 | + phy_stop(priv->phydev); | |
| 1002 | + | |
| 1003 | + netif_stop_queue(dev); | |
| 1004 | + napi_disable(&priv->napi); | |
| 1005 | + | |
| 1006 | + nb8800_dma_stop(dev); | |
| 1007 | + nb8800_mac_rx(dev, false); | |
| 1008 | + nb8800_mac_tx(dev, false); | |
| 1009 | + | |
| 1010 | + phy_disconnect(priv->phydev); | |
| 1011 | + priv->phydev = NULL; | |
| 1012 | + | |
| 1013 | + free_irq(dev->irq, dev); | |
| 1014 | + | |
| 1015 | + nb8800_dma_free(dev); | |
| 1016 | + | |
| 1017 | + return 0; | |
| 1018 | +} | |
| 1019 | + | |
| 1020 | +static int nb8800_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) | |
| 1021 | +{ | |
| 1022 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1023 | + | |
| 1024 | + return phy_mii_ioctl(priv->phydev, rq, cmd); | |
| 1025 | +} | |
| 1026 | + | |
| 1027 | +static const struct net_device_ops nb8800_netdev_ops = { | |
| 1028 | + .ndo_open = nb8800_open, | |
| 1029 | + .ndo_stop = nb8800_stop, | |
| 1030 | + .ndo_start_xmit = nb8800_xmit, | |
| 1031 | + .ndo_set_mac_address = nb8800_set_mac_address, | |
| 1032 | + .ndo_set_rx_mode = nb8800_set_rx_mode, | |
| 1033 | + .ndo_do_ioctl = nb8800_ioctl, | |
| 1034 | + .ndo_change_mtu = eth_change_mtu, | |
| 1035 | + .ndo_validate_addr = eth_validate_addr, | |
| 1036 | +}; | |
| 1037 | + | |
| 1038 | +static int nb8800_get_settings(struct net_device *dev, struct ethtool_cmd *cmd) | |
| 1039 | +{ | |
| 1040 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1041 | + | |
| 1042 | + if (!priv->phydev) | |
| 1043 | + return -ENODEV; | |
| 1044 | + | |
| 1045 | + return phy_ethtool_gset(priv->phydev, cmd); | |
| 1046 | +} | |
| 1047 | + | |
| 1048 | +static int nb8800_set_settings(struct net_device *dev, struct ethtool_cmd *cmd) | |
| 1049 | +{ | |
| 1050 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1051 | + | |
| 1052 | + if (!priv->phydev) | |
| 1053 | + return -ENODEV; | |
| 1054 | + | |
| 1055 | + return phy_ethtool_sset(priv->phydev, cmd); | |
| 1056 | +} | |
| 1057 | + | |
| 1058 | +static int nb8800_nway_reset(struct net_device *dev) | |
| 1059 | +{ | |
| 1060 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1061 | + | |
| 1062 | + if (!priv->phydev) | |
| 1063 | + return -ENODEV; | |
| 1064 | + | |
| 1065 | + return genphy_restart_aneg(priv->phydev); | |
| 1066 | +} | |
| 1067 | + | |
| 1068 | +static void nb8800_get_pauseparam(struct net_device *dev, | |
| 1069 | + struct ethtool_pauseparam *pp) | |
| 1070 | +{ | |
| 1071 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1072 | + | |
| 1073 | + pp->autoneg = priv->pause_aneg; | |
| 1074 | + pp->rx_pause = priv->pause_rx; | |
| 1075 | + pp->tx_pause = priv->pause_tx; | |
| 1076 | +} | |
| 1077 | + | |
| 1078 | +static int nb8800_set_pauseparam(struct net_device *dev, | |
| 1079 | + struct ethtool_pauseparam *pp) | |
| 1080 | +{ | |
| 1081 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1082 | + | |
| 1083 | + priv->pause_aneg = pp->autoneg; | |
| 1084 | + priv->pause_rx = pp->rx_pause; | |
| 1085 | + priv->pause_tx = pp->tx_pause; | |
| 1086 | + | |
| 1087 | + nb8800_pause_adv(dev); | |
| 1088 | + | |
| 1089 | + if (!priv->pause_aneg) | |
| 1090 | + nb8800_pause_config(dev); | |
| 1091 | + else if (priv->phydev) | |
| 1092 | + phy_start_aneg(priv->phydev); | |
| 1093 | + | |
| 1094 | + return 0; | |
| 1095 | +} | |
| 1096 | + | |
| 1097 | +static const char nb8800_stats_names[][ETH_GSTRING_LEN] = { | |
| 1098 | + "rx_bytes_ok", | |
| 1099 | + "rx_frames_ok", | |
| 1100 | + "rx_undersize_frames", | |
| 1101 | + "rx_fragment_frames", | |
| 1102 | + "rx_64_byte_frames", | |
| 1103 | + "rx_127_byte_frames", | |
| 1104 | + "rx_255_byte_frames", | |
| 1105 | + "rx_511_byte_frames", | |
| 1106 | + "rx_1023_byte_frames", | |
| 1107 | + "rx_max_size_frames", | |
| 1108 | + "rx_oversize_frames", | |
| 1109 | + "rx_bad_fcs_frames", | |
| 1110 | + "rx_broadcast_frames", | |
| 1111 | + "rx_multicast_frames", | |
| 1112 | + "rx_control_frames", | |
| 1113 | + "rx_pause_frames", | |
| 1114 | + "rx_unsup_control_frames", | |
| 1115 | + "rx_align_error_frames", | |
| 1116 | + "rx_overrun_frames", | |
| 1117 | + "rx_jabber_frames", | |
| 1118 | + "rx_bytes", | |
| 1119 | + "rx_frames", | |
| 1120 | + | |
| 1121 | + "tx_bytes_ok", | |
| 1122 | + "tx_frames_ok", | |
| 1123 | + "tx_64_byte_frames", | |
| 1124 | + "tx_127_byte_frames", | |
| 1125 | + "tx_255_byte_frames", | |
| 1126 | + "tx_511_byte_frames", | |
| 1127 | + "tx_1023_byte_frames", | |
| 1128 | + "tx_max_size_frames", | |
| 1129 | + "tx_oversize_frames", | |
| 1130 | + "tx_broadcast_frames", | |
| 1131 | + "tx_multicast_frames", | |
| 1132 | + "tx_control_frames", | |
| 1133 | + "tx_pause_frames", | |
| 1134 | + "tx_underrun_frames", | |
| 1135 | + "tx_single_collision_frames", | |
| 1136 | + "tx_multi_collision_frames", | |
| 1137 | + "tx_deferred_collision_frames", | |
| 1138 | + "tx_late_collision_frames", | |
| 1139 | + "tx_excessive_collision_frames", | |
| 1140 | + "tx_bytes", | |
| 1141 | + "tx_frames", | |
| 1142 | + "tx_collisions", | |
| 1143 | +}; | |
| 1144 | + | |
| 1145 | +#define NB8800_NUM_STATS ARRAY_SIZE(nb8800_stats_names) | |
| 1146 | + | |
| 1147 | +static int nb8800_get_sset_count(struct net_device *dev, int sset) | |
| 1148 | +{ | |
| 1149 | + if (sset == ETH_SS_STATS) | |
| 1150 | + return NB8800_NUM_STATS; | |
| 1151 | + | |
| 1152 | + return -EOPNOTSUPP; | |
| 1153 | +} | |
| 1154 | + | |
| 1155 | +static void nb8800_get_strings(struct net_device *dev, u32 sset, u8 *buf) | |
| 1156 | +{ | |
| 1157 | + if (sset == ETH_SS_STATS) | |
| 1158 | + memcpy(buf, &nb8800_stats_names, sizeof(nb8800_stats_names)); | |
| 1159 | +} | |
| 1160 | + | |
| 1161 | +static u32 nb8800_read_stat(struct net_device *dev, int index) | |
| 1162 | +{ | |
| 1163 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1164 | + | |
| 1165 | + nb8800_writeb(priv, NB8800_STAT_INDEX, index); | |
| 1166 | + | |
| 1167 | + return nb8800_readl(priv, NB8800_STAT_DATA); | |
| 1168 | +} | |
| 1169 | + | |
| 1170 | +static void nb8800_get_ethtool_stats(struct net_device *dev, | |
| 1171 | + struct ethtool_stats *estats, u64 *st) | |
| 1172 | +{ | |
| 1173 | + unsigned int i; | |
| 1174 | + u32 rx, tx; | |
| 1175 | + | |
| 1176 | + for (i = 0; i < NB8800_NUM_STATS / 2; i++) { | |
| 1177 | + rx = nb8800_read_stat(dev, i); | |
| 1178 | + tx = nb8800_read_stat(dev, i | 0x80); | |
| 1179 | + st[i] = rx; | |
| 1180 | + st[i + NB8800_NUM_STATS / 2] = tx; | |
| 1181 | + } | |
| 1182 | +} | |
| 1183 | + | |
| 1184 | +static const struct ethtool_ops nb8800_ethtool_ops = { | |
| 1185 | + .get_settings = nb8800_get_settings, | |
| 1186 | + .set_settings = nb8800_set_settings, | |
| 1187 | + .nway_reset = nb8800_nway_reset, | |
| 1188 | + .get_link = ethtool_op_get_link, | |
| 1189 | + .get_pauseparam = nb8800_get_pauseparam, | |
| 1190 | + .set_pauseparam = nb8800_set_pauseparam, | |
| 1191 | + .get_sset_count = nb8800_get_sset_count, | |
| 1192 | + .get_strings = nb8800_get_strings, | |
| 1193 | + .get_ethtool_stats = nb8800_get_ethtool_stats, | |
| 1194 | +}; | |
| 1195 | + | |
| 1196 | +static int nb8800_hw_init(struct net_device *dev) | |
| 1197 | +{ | |
| 1198 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1199 | + u32 val; | |
| 1200 | + | |
| 1201 | + val = TX_RETRY_EN | TX_PAD_EN | TX_APPEND_FCS; | |
| 1202 | + nb8800_writeb(priv, NB8800_TX_CTL1, val); | |
| 1203 | + | |
| 1204 | + /* Collision retry count */ | |
| 1205 | + nb8800_writeb(priv, NB8800_TX_CTL2, 5); | |
| 1206 | + | |
| 1207 | + val = RX_PAD_STRIP | RX_AF_EN; | |
| 1208 | + nb8800_writeb(priv, NB8800_RX_CTL, val); | |
| 1209 | + | |
| 1210 | + /* Chosen by fair dice roll */ | |
| 1211 | + nb8800_writeb(priv, NB8800_RANDOM_SEED, 4); | |
| 1212 | + | |
| 1213 | + /* TX cycles per deferral period */ | |
| 1214 | + nb8800_writeb(priv, NB8800_TX_SDP, 12); | |
| 1215 | + | |
| 1216 | + /* The following three threshold values have been | |
| 1217 | + * experimentally determined for good results. | |
| 1218 | + */ | |
| 1219 | + | |
| 1220 | + /* RX/TX FIFO threshold for partial empty (64-bit entries) */ | |
| 1221 | + nb8800_writeb(priv, NB8800_PE_THRESHOLD, 0); | |
| 1222 | + | |
| 1223 | + /* RX/TX FIFO threshold for partial full (64-bit entries) */ | |
| 1224 | + nb8800_writeb(priv, NB8800_PF_THRESHOLD, 255); | |
| 1225 | + | |
| 1226 | + /* Buffer size for transmit (64-bit entries) */ | |
| 1227 | + nb8800_writeb(priv, NB8800_TX_BUFSIZE, 64); | |
| 1228 | + | |
| 1229 | + /* Configure tx DMA */ | |
| 1230 | + | |
| 1231 | + val = nb8800_readl(priv, NB8800_TXC_CR); | |
| 1232 | + val &= TCR_LE; /* keep endian setting */ | |
| 1233 | + val |= TCR_DM; /* DMA descriptor mode */ | |
| 1234 | + val |= TCR_RS; /* automatically store tx status */ | |
| 1235 | + val |= TCR_DIE; /* interrupt on DMA chain completion */ | |
| 1236 | + val |= TCR_TFI(7); /* interrupt after 7 frames transmitted */ | |
| 1237 | + val |= TCR_BTS(2); /* 32-byte bus transaction size */ | |
| 1238 | + nb8800_writel(priv, NB8800_TXC_CR, val); | |
| 1239 | + | |
| 1240 | + /* TX complete interrupt after 10 ms or 7 frames (see above) */ | |
| 1241 | + val = clk_get_rate(priv->clk) / 100; | |
| 1242 | + nb8800_writel(priv, NB8800_TX_ITR, val); | |
| 1243 | + | |
| 1244 | + /* Configure rx DMA */ | |
| 1245 | + | |
| 1246 | + val = nb8800_readl(priv, NB8800_RXC_CR); | |
| 1247 | + val &= RCR_LE; /* keep endian setting */ | |
| 1248 | + val |= RCR_DM; /* DMA descriptor mode */ | |
| 1249 | + val |= RCR_RS; /* automatically store rx status */ | |
| 1250 | + val |= RCR_DIE; /* interrupt at end of DMA chain */ | |
| 1251 | + val |= RCR_RFI(7); /* interrupt after 7 frames received */ | |
| 1252 | + val |= RCR_BTS(2); /* 32-byte bus transaction size */ | |
| 1253 | + nb8800_writel(priv, NB8800_RXC_CR, val); | |
| 1254 | + | |
| 1255 | + /* The rx interrupt can fire before the DMA has completed | |
| 1256 | + * unless a small delay is added. 50 us is hopefully enough. | |
| 1257 | + */ | |
| 1258 | + priv->rx_itr_irq = clk_get_rate(priv->clk) / 20000; | |
| 1259 | + | |
| 1260 | + /* In NAPI poll mode we want to disable interrupts, but the | |
| 1261 | + * hardware does not permit this. Delay 10 ms instead. | |
| 1262 | + */ | |
| 1263 | + priv->rx_itr_poll = clk_get_rate(priv->clk) / 100; | |
| 1264 | + | |
| 1265 | + nb8800_writel(priv, NB8800_RX_ITR, priv->rx_itr_irq); | |
| 1266 | + | |
| 1267 | + priv->rx_dma_config = RX_BUF_SIZE | DESC_BTS(2) | DESC_DS | DESC_EOF; | |
| 1268 | + | |
| 1269 | + /* Flow control settings */ | |
| 1270 | + | |
| 1271 | + /* Pause time of 0.1 ms */ | |
| 1272 | + val = 100000 / 512; | |
| 1273 | + nb8800_writeb(priv, NB8800_PQ1, val >> 8); | |
| 1274 | + nb8800_writeb(priv, NB8800_PQ2, val & 0xff); | |
| 1275 | + | |
| 1276 | + /* Auto-negotiate by default */ | |
| 1277 | + priv->pause_aneg = true; | |
| 1278 | + priv->pause_rx = true; | |
| 1279 | + priv->pause_tx = true; | |
| 1280 | + | |
| 1281 | + nb8800_mc_init(dev, 0); | |
| 1282 | + | |
| 1283 | + return 0; | |
| 1284 | +} | |
| 1285 | + | |
| 1286 | +static int nb8800_tangox_init(struct net_device *dev) | |
| 1287 | +{ | |
| 1288 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1289 | + u32 pad_mode = PAD_MODE_MII; | |
| 1290 | + | |
| 1291 | + switch (priv->phy_mode) { | |
| 1292 | + case PHY_INTERFACE_MODE_MII: | |
| 1293 | + case PHY_INTERFACE_MODE_GMII: | |
| 1294 | + pad_mode = PAD_MODE_MII; | |
| 1295 | + break; | |
| 1296 | + | |
| 1297 | + case PHY_INTERFACE_MODE_RGMII: | |
| 1298 | + pad_mode = PAD_MODE_RGMII; | |
| 1299 | + break; | |
| 1300 | + | |
| 1301 | + case PHY_INTERFACE_MODE_RGMII_TXID: | |
| 1302 | + pad_mode = PAD_MODE_RGMII | PAD_MODE_GTX_CLK_DELAY; | |
| 1303 | + break; | |
| 1304 | + | |
| 1305 | + default: | |
| 1306 | + dev_err(dev->dev.parent, "unsupported phy mode %s\n", | |
| 1307 | + phy_modes(priv->phy_mode)); | |
| 1308 | + return -EINVAL; | |
| 1309 | + } | |
| 1310 | + | |
| 1311 | + nb8800_writeb(priv, NB8800_TANGOX_PAD_MODE, pad_mode); | |
| 1312 | + | |
| 1313 | + return 0; | |
| 1314 | +} | |
| 1315 | + | |
| 1316 | +static int nb8800_tangox_reset(struct net_device *dev) | |
| 1317 | +{ | |
| 1318 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1319 | + int clk_div; | |
| 1320 | + | |
| 1321 | + nb8800_writeb(priv, NB8800_TANGOX_RESET, 0); | |
| 1322 | + usleep_range(1000, 10000); | |
| 1323 | + nb8800_writeb(priv, NB8800_TANGOX_RESET, 1); | |
| 1324 | + | |
| 1325 | + wmb(); /* ensure reset is cleared before proceeding */ | |
| 1326 | + | |
| 1327 | + clk_div = DIV_ROUND_UP(clk_get_rate(priv->clk), 2 * MAX_MDC_CLOCK); | |
| 1328 | + nb8800_writew(priv, NB8800_TANGOX_MDIO_CLKDIV, clk_div); | |
| 1329 | + | |
| 1330 | + return 0; | |
| 1331 | +} | |
| 1332 | + | |
| 1333 | +static const struct nb8800_ops nb8800_tangox_ops = { | |
| 1334 | + .init = nb8800_tangox_init, | |
| 1335 | + .reset = nb8800_tangox_reset, | |
| 1336 | +}; | |
| 1337 | + | |
| 1338 | +static int nb8800_tango4_init(struct net_device *dev) | |
| 1339 | +{ | |
| 1340 | + struct nb8800_priv *priv = netdev_priv(dev); | |
| 1341 | + int err; | |
| 1342 | + | |
| 1343 | + err = nb8800_tangox_init(dev); | |
| 1344 | + if (err) | |
| 1345 | + return err; | |
| 1346 | + | |
| 1347 | + /* On tango4 interrupt on DMA completion per frame works and gives | |
| 1348 | + * better performance despite generating more rx interrupts. | |
| 1349 | + */ | |
| 1350 | + | |
| 1351 | + /* Disable unnecessary interrupt on rx completion */ | |
| 1352 | + nb8800_clearl(priv, NB8800_RXC_CR, RCR_RFI(7)); | |
| 1353 | + | |
| 1354 | + /* Request interrupt on descriptor DMA completion */ | |
| 1355 | + priv->rx_dma_config |= DESC_ID; | |
| 1356 | + | |
| 1357 | + return 0; | |
| 1358 | +} | |
| 1359 | + | |
| 1360 | +static const struct nb8800_ops nb8800_tango4_ops = { | |
| 1361 | + .init = nb8800_tango4_init, | |
| 1362 | + .reset = nb8800_tangox_reset, | |
| 1363 | +}; | |
| 1364 | + | |
| 1365 | +static const struct of_device_id nb8800_dt_ids[] = { | |
| 1366 | + { | |
| 1367 | + .compatible = "aurora,nb8800", | |
| 1368 | + }, | |
| 1369 | + { | |
| 1370 | + .compatible = "sigma,smp8642-ethernet", | |
| 1371 | + .data = &nb8800_tangox_ops, | |
| 1372 | + }, | |
| 1373 | + { | |
| 1374 | + .compatible = "sigma,smp8734-ethernet", | |
| 1375 | + .data = &nb8800_tango4_ops, | |
| 1376 | + }, | |
| 1377 | + { } | |
| 1378 | +}; | |
| 1379 | + | |
| 1380 | +static int nb8800_probe(struct platform_device *pdev) | |
| 1381 | +{ | |
| 1382 | + const struct of_device_id *match; | |
| 1383 | + const struct nb8800_ops *ops = NULL; | |
| 1384 | + struct nb8800_priv *priv; | |
| 1385 | + struct resource *res; | |
| 1386 | + struct net_device *dev; | |
| 1387 | + struct mii_bus *bus; | |
| 1388 | + const unsigned char *mac; | |
| 1389 | + void __iomem *base; | |
| 1390 | + int irq; | |
| 1391 | + int ret; | |
| 1392 | + | |
| 1393 | + match = of_match_device(nb8800_dt_ids, &pdev->dev); | |
| 1394 | + if (match) | |
| 1395 | + ops = match->data; | |
| 1396 | + | |
| 1397 | + irq = platform_get_irq(pdev, 0); | |
| 1398 | + if (irq <= 0) { | |
| 1399 | + dev_err(&pdev->dev, "No IRQ\n"); | |
| 1400 | + return -EINVAL; | |
| 1401 | + } | |
| 1402 | + | |
| 1403 | + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); | |
| 1404 | + base = devm_ioremap_resource(&pdev->dev, res); | |
| 1405 | + if (IS_ERR(base)) | |
| 1406 | + return PTR_ERR(base); | |
| 1407 | + | |
| 1408 | + dev_dbg(&pdev->dev, "AU-NB8800 Ethernet at %pa\n", &res->start); | |
| 1409 | + | |
| 1410 | + dev = alloc_etherdev(sizeof(*priv)); | |
| 1411 | + if (!dev) | |
| 1412 | + return -ENOMEM; | |
| 1413 | + | |
| 1414 | + platform_set_drvdata(pdev, dev); | |
| 1415 | + SET_NETDEV_DEV(dev, &pdev->dev); | |
| 1416 | + | |
| 1417 | + priv = netdev_priv(dev); | |
| 1418 | + priv->base = base; | |
| 1419 | + | |
| 1420 | + priv->phy_mode = of_get_phy_mode(pdev->dev.of_node); | |
| 1421 | + if (priv->phy_mode < 0) | |
| 1422 | + priv->phy_mode = PHY_INTERFACE_MODE_RGMII; | |
| 1423 | + | |
| 1424 | + priv->clk = devm_clk_get(&pdev->dev, NULL); | |
| 1425 | + if (IS_ERR(priv->clk)) { | |
| 1426 | + dev_err(&pdev->dev, "failed to get clock\n"); | |
| 1427 | + ret = PTR_ERR(priv->clk); | |
| 1428 | + goto err_free_dev; | |
| 1429 | + } | |
| 1430 | + | |
| 1431 | + ret = clk_prepare_enable(priv->clk); | |
| 1432 | + if (ret) | |
| 1433 | + goto err_free_dev; | |
| 1434 | + | |
| 1435 | + spin_lock_init(&priv->tx_lock); | |
| 1436 | + | |
| 1437 | + if (ops && ops->reset) { | |
| 1438 | + ret = ops->reset(dev); | |
| 1439 | + if (ret) | |
| 1440 | + goto err_free_dev; | |
| 1441 | + } | |
| 1442 | + | |
| 1443 | + bus = devm_mdiobus_alloc(&pdev->dev); | |
| 1444 | + if (!bus) { | |
| 1445 | + ret = -ENOMEM; | |
| 1446 | + goto err_disable_clk; | |
| 1447 | + } | |
| 1448 | + | |
| 1449 | + bus->name = "nb8800-mii"; | |
| 1450 | + bus->read = nb8800_mdio_read; | |
| 1451 | + bus->write = nb8800_mdio_write; | |
| 1452 | + bus->parent = &pdev->dev; | |
| 1453 | + snprintf(bus->id, MII_BUS_ID_SIZE, "%lx.nb8800-mii", | |
| 1454 | + (unsigned long)res->start); | |
| 1455 | + bus->priv = priv; | |
| 1456 | + | |
| 1457 | + ret = of_mdiobus_register(bus, pdev->dev.of_node); | |
| 1458 | + if (ret) { | |
| 1459 | + dev_err(&pdev->dev, "failed to register MII bus\n"); | |
| 1460 | + goto err_disable_clk; | |
| 1461 | + } | |
| 1462 | + | |
| 1463 | + priv->phy_node = of_parse_phandle(pdev->dev.of_node, "phy-handle", 0); | |
| 1464 | + if (!priv->phy_node) { | |
| 1465 | + dev_err(&pdev->dev, "no PHY specified\n"); | |
| 1466 | + ret = -ENODEV; | |
| 1467 | + goto err_free_bus; | |
| 1468 | + } | |
| 1469 | + | |
| 1470 | + priv->mii_bus = bus; | |
| 1471 | + | |
| 1472 | + ret = nb8800_hw_init(dev); | |
| 1473 | + if (ret) | |
| 1474 | + goto err_free_bus; | |
| 1475 | + | |
| 1476 | + if (ops && ops->init) { | |
| 1477 | + ret = ops->init(dev); | |
| 1478 | + if (ret) | |
| 1479 | + goto err_free_bus; | |
| 1480 | + } | |
| 1481 | + | |
| 1482 | + dev->netdev_ops = &nb8800_netdev_ops; | |
| 1483 | + dev->ethtool_ops = &nb8800_ethtool_ops; | |
| 1484 | + dev->flags |= IFF_MULTICAST; | |
| 1485 | + dev->irq = irq; | |
| 1486 | + | |
| 1487 | + mac = of_get_mac_address(pdev->dev.of_node); | |
| 1488 | + if (mac) | |
| 1489 | + ether_addr_copy(dev->dev_addr, mac); | |
| 1490 | + | |
| 1491 | + if (!is_valid_ether_addr(dev->dev_addr)) | |
| 1492 | + eth_hw_addr_random(dev); | |
| 1493 | + | |
| 1494 | + nb8800_update_mac_addr(dev); | |
| 1495 | + | |
| 1496 | + netif_carrier_off(dev); | |
| 1497 | + | |
| 1498 | + ret = register_netdev(dev); | |
| 1499 | + if (ret) { | |
| 1500 | + netdev_err(dev, "failed to register netdev\n"); | |
| 1501 | + goto err_free_dma; | |
| 1502 | + } | |
| 1503 | + | |
| 1504 | + netif_napi_add(dev, &priv->napi, nb8800_poll, NAPI_POLL_WEIGHT); | |
| 1505 | + | |
| 1506 | + netdev_info(dev, "MAC address %pM\n", dev->dev_addr); | |
| 1507 | + | |
| 1508 | + return 0; | |
| 1509 | + | |
| 1510 | +err_free_dma: | |
| 1511 | + nb8800_dma_free(dev); | |
| 1512 | +err_free_bus: | |
| 1513 | + mdiobus_unregister(bus); | |
| 1514 | +err_disable_clk: | |
| 1515 | + clk_disable_unprepare(priv->clk); | |
| 1516 | +err_free_dev: | |
| 1517 | + free_netdev(dev); | |
| 1518 | + | |
| 1519 | + return ret; | |
| 1520 | +} | |
| 1521 | + | |
| 1522 | +static int nb8800_remove(struct platform_device *pdev) | |
| 1523 | +{ | |
| 1524 | + struct net_device *ndev = platform_get_drvdata(pdev); | |
| 1525 | + struct nb8800_priv *priv = netdev_priv(ndev); | |
| 1526 | + | |
| 1527 | + unregister_netdev(ndev); | |
| 1528 | + | |
| 1529 | + mdiobus_unregister(priv->mii_bus); | |
| 1530 | + | |
| 1531 | + clk_disable_unprepare(priv->clk); | |
| 1532 | + | |
| 1533 | + nb8800_dma_free(ndev); | |
| 1534 | + free_netdev(ndev); | |
| 1535 | + | |
| 1536 | + return 0; | |
| 1537 | +} | |
| 1538 | + | |
| 1539 | +static struct platform_driver nb8800_driver = { | |
| 1540 | + .driver = { | |
| 1541 | + .name = "nb8800", | |
| 1542 | + .of_match_table = nb8800_dt_ids, | |
| 1543 | + }, | |
| 1544 | + .probe = nb8800_probe, | |
| 1545 | + .remove = nb8800_remove, | |
| 1546 | +}; | |
| 1547 | + | |
| 1548 | +module_platform_driver(nb8800_driver); | |
| 1549 | + | |
| 1550 | +MODULE_DESCRIPTION("Aurora AU-NB8800 Ethernet driver"); | |
| 1551 | +MODULE_AUTHOR("Mans Rullgard <mans@mansr.com>"); | |
| 1552 | +MODULE_LICENSE("GPL"); | |
| @@ -0,0 +1,316 @@ | ||
| 1 | +#ifndef _NB8800_H_ | |
| 2 | +#define _NB8800_H_ | |
| 3 | + | |
| 4 | +#include <linux/types.h> | |
| 5 | +#include <linux/skbuff.h> | |
| 6 | +#include <linux/phy.h> | |
| 7 | +#include <linux/clk.h> | |
| 8 | +#include <linux/bitops.h> | |
| 9 | + | |
| 10 | +#define RX_DESC_COUNT 256 | |
| 11 | +#define TX_DESC_COUNT 256 | |
| 12 | + | |
| 13 | +#define NB8800_DESC_LOW 4 | |
| 14 | + | |
| 15 | +#define RX_BUF_SIZE 1552 | |
| 16 | + | |
| 17 | +#define RX_COPYBREAK 256 | |
| 18 | +#define RX_COPYHDR 128 | |
| 19 | + | |
| 20 | +#define MAX_MDC_CLOCK 2500000 | |
| 21 | + | |
| 22 | +/* Stargate Solutions SSN8800 core registers */ | |
| 23 | +#define NB8800_TX_CTL1 0x000 | |
| 24 | +#define TX_TPD BIT(5) | |
| 25 | +#define TX_APPEND_FCS BIT(4) | |
| 26 | +#define TX_PAD_EN BIT(3) | |
| 27 | +#define TX_RETRY_EN BIT(2) | |
| 28 | +#define TX_EN BIT(0) | |
| 29 | + | |
| 30 | +#define NB8800_TX_CTL2 0x001 | |
| 31 | + | |
| 32 | +#define NB8800_RX_CTL 0x004 | |
| 33 | +#define RX_BC_DISABLE BIT(7) | |
| 34 | +#define RX_RUNT BIT(6) | |
| 35 | +#define RX_AF_EN BIT(5) | |
| 36 | +#define RX_PAUSE_EN BIT(3) | |
| 37 | +#define RX_SEND_CRC BIT(2) | |
| 38 | +#define RX_PAD_STRIP BIT(1) | |
| 39 | +#define RX_EN BIT(0) | |
| 40 | + | |
| 41 | +#define NB8800_RANDOM_SEED 0x008 | |
| 42 | +#define NB8800_TX_SDP 0x14 | |
| 43 | +#define NB8800_TX_TPDP1 0x18 | |
| 44 | +#define NB8800_TX_TPDP2 0x19 | |
| 45 | +#define NB8800_SLOT_TIME 0x1c | |
| 46 | + | |
| 47 | +#define NB8800_MDIO_CMD 0x020 | |
| 48 | +#define MDIO_CMD_GO BIT(31) | |
| 49 | +#define MDIO_CMD_WR BIT(26) | |
| 50 | +#define MDIO_CMD_ADDR(x) ((x) << 21) | |
| 51 | +#define MDIO_CMD_REG(x) ((x) << 16) | |
| 52 | +#define MDIO_CMD_DATA(x) ((x) << 0) | |
| 53 | + | |
| 54 | +#define NB8800_MDIO_STS 0x024 | |
| 55 | +#define MDIO_STS_ERR BIT(31) | |
| 56 | + | |
| 57 | +#define NB8800_MC_ADDR(i) (0x028 + (i)) | |
| 58 | +#define NB8800_MC_INIT 0x02e | |
| 59 | +#define NB8800_UC_ADDR(i) (0x03c + (i)) | |
| 60 | + | |
| 61 | +#define NB8800_MAC_MODE 0x044 | |
| 62 | +#define RGMII_MODE BIT(7) | |
| 63 | +#define HALF_DUPLEX BIT(4) | |
| 64 | +#define BURST_EN BIT(3) | |
| 65 | +#define LOOPBACK_EN BIT(2) | |
| 66 | +#define GMAC_MODE BIT(0) | |
| 67 | + | |
| 68 | +#define NB8800_IC_THRESHOLD 0x050 | |
| 69 | +#define NB8800_PE_THRESHOLD 0x051 | |
| 70 | +#define NB8800_PF_THRESHOLD 0x052 | |
| 71 | +#define NB8800_TX_BUFSIZE 0x054 | |
| 72 | +#define NB8800_FIFO_CTL 0x056 | |
| 73 | +#define NB8800_PQ1 0x060 | |
| 74 | +#define NB8800_PQ2 0x061 | |
| 75 | +#define NB8800_SRC_ADDR(i) (0x06a + (i)) | |
| 76 | +#define NB8800_STAT_DATA 0x078 | |
| 77 | +#define NB8800_STAT_INDEX 0x07c | |
| 78 | +#define NB8800_STAT_CLEAR 0x07d | |
| 79 | + | |
| 80 | +#define NB8800_SLEEP_MODE 0x07e | |
| 81 | +#define SLEEP_MODE BIT(0) | |
| 82 | + | |
| 83 | +#define NB8800_WAKEUP 0x07f | |
| 84 | +#define WAKEUP BIT(0) | |
| 85 | + | |
| 86 | +/* Aurora NB8800 host interface registers */ | |
| 87 | +#define NB8800_TXC_CR 0x100 | |
| 88 | +#define TCR_LK BIT(12) | |
| 89 | +#define TCR_DS BIT(11) | |
| 90 | +#define TCR_BTS(x) (((x) & 0x7) << 8) | |
| 91 | +#define TCR_DIE BIT(7) | |
| 92 | +#define TCR_TFI(x) (((x) & 0x7) << 4) | |
| 93 | +#define TCR_LE BIT(3) | |
| 94 | +#define TCR_RS BIT(2) | |
| 95 | +#define TCR_DM BIT(1) | |
| 96 | +#define TCR_EN BIT(0) | |
| 97 | + | |
| 98 | +#define NB8800_TXC_SR 0x104 | |
| 99 | +#define TSR_DE BIT(3) | |
| 100 | +#define TSR_DI BIT(2) | |
| 101 | +#define TSR_TO BIT(1) | |
| 102 | +#define TSR_TI BIT(0) | |
| 103 | + | |
| 104 | +#define NB8800_TX_SAR 0x108 | |
| 105 | +#define NB8800_TX_DESC_ADDR 0x10c | |
| 106 | + | |
| 107 | +#define NB8800_TX_REPORT_ADDR 0x110 | |
| 108 | +#define TX_BYTES_TRANSFERRED(x) (((x) >> 16) & 0xffff) | |
| 109 | +#define TX_FIRST_DEFERRAL BIT(7) | |
| 110 | +#define TX_EARLY_COLLISIONS(x) (((x) >> 3) & 0xf) | |
| 111 | +#define TX_LATE_COLLISION BIT(2) | |
| 112 | +#define TX_PACKET_DROPPED BIT(1) | |
| 113 | +#define TX_FIFO_UNDERRUN BIT(0) | |
| 114 | +#define IS_TX_ERROR(r) ((r) & 0x07) | |
| 115 | + | |
| 116 | +#define NB8800_TX_FIFO_SR 0x114 | |
| 117 | +#define NB8800_TX_ITR 0x118 | |
| 118 | + | |
| 119 | +#define NB8800_RXC_CR 0x200 | |
| 120 | +#define RCR_FL BIT(13) | |
| 121 | +#define RCR_LK BIT(12) | |
| 122 | +#define RCR_DS BIT(11) | |
| 123 | +#define RCR_BTS(x) (((x) & 7) << 8) | |
| 124 | +#define RCR_DIE BIT(7) | |
| 125 | +#define RCR_RFI(x) (((x) & 7) << 4) | |
| 126 | +#define RCR_LE BIT(3) | |
| 127 | +#define RCR_RS BIT(2) | |
| 128 | +#define RCR_DM BIT(1) | |
| 129 | +#define RCR_EN BIT(0) | |
| 130 | + | |
| 131 | +#define NB8800_RXC_SR 0x204 | |
| 132 | +#define RSR_DE BIT(3) | |
| 133 | +#define RSR_DI BIT(2) | |
| 134 | +#define RSR_RO BIT(1) | |
| 135 | +#define RSR_RI BIT(0) | |
| 136 | + | |
| 137 | +#define NB8800_RX_SAR 0x208 | |
| 138 | +#define NB8800_RX_DESC_ADDR 0x20c | |
| 139 | + | |
| 140 | +#define NB8800_RX_REPORT_ADDR 0x210 | |
| 141 | +#define RX_BYTES_TRANSFERRED(x) (((x) >> 16) & 0xFFFF) | |
| 142 | +#define RX_MULTICAST_PKT BIT(9) | |
| 143 | +#define RX_BROADCAST_PKT BIT(8) | |
| 144 | +#define RX_LENGTH_ERR BIT(7) | |
| 145 | +#define RX_FCS_ERR BIT(6) | |
| 146 | +#define RX_RUNT_PKT BIT(5) | |
| 147 | +#define RX_FIFO_OVERRUN BIT(4) | |
| 148 | +#define RX_LATE_COLLISION BIT(3) | |
| 149 | +#define RX_ALIGNMENT_ERROR BIT(2) | |
| 150 | +#define RX_ERROR_MASK 0xfc | |
| 151 | +#define IS_RX_ERROR(r) ((r) & RX_ERROR_MASK) | |
| 152 | + | |
| 153 | +#define NB8800_RX_FIFO_SR 0x214 | |
| 154 | +#define NB8800_RX_ITR 0x218 | |
| 155 | + | |
| 156 | +/* Sigma Designs SMP86xx additional registers */ | |
| 157 | +#define NB8800_TANGOX_PAD_MODE 0x400 | |
| 158 | +#define PAD_MODE_MASK 0x7 | |
| 159 | +#define PAD_MODE_MII 0x0 | |
| 160 | +#define PAD_MODE_RGMII 0x1 | |
| 161 | +#define PAD_MODE_GTX_CLK_INV BIT(3) | |
| 162 | +#define PAD_MODE_GTX_CLK_DELAY BIT(4) | |
| 163 | + | |
| 164 | +#define NB8800_TANGOX_MDIO_CLKDIV 0x420 | |
| 165 | +#define NB8800_TANGOX_RESET 0x424 | |
| 166 | + | |
| 167 | +/* Hardware DMA descriptor */ | |
| 168 | +struct nb8800_dma_desc { | |
| 169 | + u32 s_addr; /* start address */ | |
| 170 | + u32 n_addr; /* next descriptor address */ | |
| 171 | + u32 r_addr; /* report address */ | |
| 172 | + u32 config; | |
| 173 | +} __aligned(8); | |
| 174 | + | |
| 175 | +#define DESC_ID BIT(23) | |
| 176 | +#define DESC_EOC BIT(22) | |
| 177 | +#define DESC_EOF BIT(21) | |
| 178 | +#define DESC_LK BIT(20) | |
| 179 | +#define DESC_DS BIT(19) | |
| 180 | +#define DESC_BTS(x) (((x) & 0x7) << 16) | |
| 181 | + | |
| 182 | +/* DMA descriptor and associated data for rx. | |
| 183 | + * Allocated from coherent memory. | |
| 184 | + */ | |
| 185 | +struct nb8800_rx_desc { | |
| 186 | + /* DMA descriptor */ | |
| 187 | + struct nb8800_dma_desc desc; | |
| 188 | + | |
| 189 | + /* Status report filled in by hardware */ | |
| 190 | + u32 report; | |
| 191 | +}; | |
| 192 | + | |
| 193 | +/* Address of buffer on rx ring */ | |
| 194 | +struct nb8800_rx_buf { | |
| 195 | + struct page *page; | |
| 196 | + unsigned long offset; | |
| 197 | +}; | |
| 198 | + | |
| 199 | +/* DMA descriptors and associated data for tx. | |
| 200 | + * Allocated from coherent memory. | |
| 201 | + */ | |
| 202 | +struct nb8800_tx_desc { | |
| 203 | + /* DMA descriptor. The second descriptor is used if packet | |
| 204 | + * data is unaligned. | |
| 205 | + */ | |
| 206 | + struct nb8800_dma_desc desc[2]; | |
| 207 | + | |
| 208 | + /* Status report filled in by hardware */ | |
| 209 | + u32 report; | |
| 210 | + | |
| 211 | + /* Bounce buffer for initial unaligned part of packet */ | |
| 212 | + u8 buf[8] __aligned(8); | |
| 213 | +}; | |
| 214 | + | |
| 215 | +/* Packet in tx queue */ | |
| 216 | +struct nb8800_tx_buf { | |
| 217 | + /* Currently queued skb */ | |
| 218 | + struct sk_buff *skb; | |
| 219 | + | |
| 220 | + /* DMA address of the first descriptor */ | |
| 221 | + dma_addr_t dma_desc; | |
| 222 | + | |
| 223 | + /* DMA address of packet data */ | |
| 224 | + dma_addr_t dma_addr; | |
| 225 | + | |
| 226 | + /* Length of DMA mapping, less than skb->len if alignment | |
| 227 | + * buffer is used. | |
| 228 | + */ | |
| 229 | + unsigned int dma_len; | |
| 230 | + | |
| 231 | + /* Number of packets in chain starting here */ | |
| 232 | + unsigned int chain_len; | |
| 233 | + | |
| 234 | + /* Packet chain ready to be submitted to hardware */ | |
| 235 | + bool ready; | |
| 236 | +}; | |
| 237 | + | |
| 238 | +struct nb8800_priv { | |
| 239 | + struct napi_struct napi; | |
| 240 | + | |
| 241 | + void __iomem *base; | |
| 242 | + | |
| 243 | + /* RX DMA descriptors */ | |
| 244 | + struct nb8800_rx_desc *rx_descs; | |
| 245 | + | |
| 246 | + /* RX buffers referenced by DMA descriptors */ | |
| 247 | + struct nb8800_rx_buf *rx_bufs; | |
| 248 | + | |
| 249 | + /* Current end of chain */ | |
| 250 | + u32 rx_eoc; | |
| 251 | + | |
| 252 | + /* Value for rx interrupt time register in NAPI interrupt mode */ | |
| 253 | + u32 rx_itr_irq; | |
| 254 | + | |
| 255 | + /* Value for rx interrupt time register in NAPI poll mode */ | |
| 256 | + u32 rx_itr_poll; | |
| 257 | + | |
| 258 | + /* Value for config field of rx DMA descriptors */ | |
| 259 | + u32 rx_dma_config; | |
| 260 | + | |
| 261 | + /* TX DMA descriptors */ | |
| 262 | + struct nb8800_tx_desc *tx_descs; | |
| 263 | + | |
| 264 | + /* TX packet queue */ | |
| 265 | + struct nb8800_tx_buf *tx_bufs; | |
| 266 | + | |
| 267 | + /* Number of free tx queue entries */ | |
| 268 | + atomic_t tx_free; | |
| 269 | + | |
| 270 | + /* First free tx queue entry */ | |
| 271 | + u32 tx_next; | |
| 272 | + | |
| 273 | + /* Next buffer to transmit */ | |
| 274 | + u32 tx_queue; | |
| 275 | + | |
| 276 | + /* Start of current packet chain */ | |
| 277 | + struct nb8800_tx_buf *tx_chain; | |
| 278 | + | |
| 279 | + /* Next buffer to reclaim */ | |
| 280 | + u32 tx_done; | |
| 281 | + | |
| 282 | + /* Lock for DMA activation */ | |
| 283 | + spinlock_t tx_lock; | |
| 284 | + | |
| 285 | + struct mii_bus *mii_bus; | |
| 286 | + struct device_node *phy_node; | |
| 287 | + struct phy_device *phydev; | |
| 288 | + | |
| 289 | + /* PHY connection type from DT */ | |
| 290 | + int phy_mode; | |
| 291 | + | |
| 292 | + /* Current link status */ | |
| 293 | + int speed; | |
| 294 | + int duplex; | |
| 295 | + int link; | |
| 296 | + | |
| 297 | + /* Pause settings */ | |
| 298 | + bool pause_aneg; | |
| 299 | + bool pause_rx; | |
| 300 | + bool pause_tx; | |
| 301 | + | |
| 302 | + /* DMA base address of rx descriptors, see rx_descs above */ | |
| 303 | + dma_addr_t rx_desc_dma; | |
| 304 | + | |
| 305 | + /* DMA base address of tx descriptors, see tx_descs above */ | |
| 306 | + dma_addr_t tx_desc_dma; | |
| 307 | + | |
| 308 | + struct clk *clk; | |
| 309 | +}; | |
| 310 | + | |
| 311 | +struct nb8800_ops { | |
| 312 | + int (*init)(struct net_device *dev); | |
| 313 | + int (*reset)(struct net_device *dev); | |
| 314 | +}; | |
| 315 | + | |
| 316 | +#endif /* _NB8800_H_ */ | |
| @@ -10139,8 +10139,8 @@ static void __bnx2x_del_vxlan_port(struct bnx2x *bp, u16 port) · 10138 unmodified lines | ||
| 10139 | 10139 | DP(BNX2X_MSG_SP, "Invalid vxlan port\n"); |
| 10140 | 10140 | return; |
| 10141 | 10141 | } |
| 10142 | − bp->vxlan_dst_port--; | |
| 10143 | − if (bp->vxlan_dst_port) | |
| 10142 | + bp->vxlan_dst_port_count--; | |
| 10143 | + if (bp->vxlan_dst_port_count) | |
| 10144 | 10144 | return; |
| 10145 | 10145 | |
| 10146 | 10146 | if (netif_running(bp->dev)) { |
| @@ -3625,6 +3625,7 @@ static int bnxt_hwrm_func_qcaps(struct bnxt *bp) · 3624 unmodified lines | ||
| 3625 | 3625 | pf->fw_fid = le16_to_cpu(resp->fid); |
| 3626 | 3626 | pf->port_id = le16_to_cpu(resp->port_id); |
| 3627 | 3627 | memcpy(pf->mac_addr, resp->perm_mac_address, ETH_ALEN); |
| 3628 | + memcpy(bp->dev->dev_addr, pf->mac_addr, ETH_ALEN); | |
| 3628 | 3629 | pf->max_rsscos_ctxs = le16_to_cpu(resp->max_rsscos_ctx); |
| 3629 | 3630 | pf->max_cp_rings = le16_to_cpu(resp->max_cmpl_rings); |
| 3630 | 3631 | pf->max_tx_rings = le16_to_cpu(resp->max_tx_rings); |
| @@ -3648,8 +3649,11 @@ static int bnxt_hwrm_func_qcaps(struct bnxt *bp) · 17 unmodified lines | ||
| 3648 | 3649 | |
| 3649 | 3650 | vf->fw_fid = le16_to_cpu(resp->fid); |
| 3650 | 3651 | memcpy(vf->mac_addr, resp->perm_mac_address, ETH_ALEN); |
| 3651 | − if (!is_valid_ether_addr(vf->mac_addr)) | |
| 3652 | − random_ether_addr(vf->mac_addr); | |
| 3652 | + if (is_valid_ether_addr(vf->mac_addr)) | |
| 3653 | + /* overwrite netdev dev_adr with admin VF MAC */ | |
| 3654 | + memcpy(bp->dev->dev_addr, vf->mac_addr, ETH_ALEN); | |
| 3655 | + else | |
| 3656 | + random_ether_addr(bp->dev->dev_addr); | |
| 3653 | 3657 | |
| 3654 | 3658 | vf->max_rsscos_ctxs = le16_to_cpu(resp->max_rsscos_ctx); |
| 3655 | 3659 | vf->max_cp_rings = le16_to_cpu(resp->max_cmpl_rings); |
| @@ -3880,6 +3884,8 @@ static int bnxt_alloc_rfs_vnics(struct bnxt *bp) · 224 unmodified lines | ||
| 3880 | 3884 | #endif |
| 3881 | 3885 | } |
| 3882 | 3886 | |
| 3887 | +static int bnxt_cfg_rx_mode(struct bnxt *); | |
| 3888 | + | |
| 3883 | 3889 | static int bnxt_init_chip(struct bnxt *bp, bool irq_re_init) |
| 3884 | 3890 | { |
| 3885 | 3891 | int rc = 0; |
| @@ -3946,11 +3952,9 @@ static int bnxt_init_chip(struct bnxt *bp, bool irq_re_init) · 60 unmodified lines | ||
| 3946 | 3952 | bp->vnic_info[0].rx_mask |= |
| 3947 | 3953 | CFA_L2_SET_RX_MASK_REQ_MASK_PROMISCUOUS; |
| 3948 | 3954 | |
| 3949 | − rc = bnxt_hwrm_cfa_l2_set_rx_mask(bp, 0); | |
| 3950 | − if (rc) { | |
| 3951 | − netdev_err(bp->dev, "HWRM cfa l2 rx mask failure rc: %x\n", rc); | |
| 3955 | + rc = bnxt_cfg_rx_mode(bp); | |
| 3956 | + if (rc) | |
| 3952 | 3957 | goto err_out; |
| 3953 | − } | |
| 3954 | 3958 | |
| 3955 | 3959 | rc = bnxt_hwrm_set_coal(bp); |
| 3956 | 3960 | if (rc) |
| @@ -4865,7 +4869,7 @@ static void bnxt_set_rx_mode(struct net_device *dev) · 908 unmodified lines | ||
| 4865 | 4869 | } |
| 4866 | 4870 | } |
| 4867 | 4871 | |
| 4868 | −static void bnxt_cfg_rx_mode(struct bnxt *bp) | |
| 4872 | +static int bnxt_cfg_rx_mode(struct bnxt *bp) | |
| 4869 | 4873 | { |
| 4870 | 4874 | struct net_device *dev = bp->dev; |
| 4871 | 4875 | struct bnxt_vnic_info *vnic = &bp->vnic_info[0]; |
| @@ -4914,6 +4918,7 @@ static void bnxt_cfg_rx_mode(struct bnxt *bp) · 42 unmodified lines | ||
| 4914 | 4918 | netdev_err(bp->dev, "HWRM vnic filter failure rc: %x\n", |
| 4915 | 4919 | rc); |
| 4916 | 4920 | vnic->uc_filter_count = i; |
| 4921 | + return rc; | |
| 4917 | 4922 | } |
| 4918 | 4923 | } |
| 4919 | 4924 | |
| @@ -4922,6 +4927,8 @@ skip_uc: · 2 unmodified lines | ||
| 4922 | 4927 | if (rc) |
| 4923 | 4928 | netdev_err(bp->dev, "HWRM cfa l2 rx mask failure rc: %x\n", |
| 4924 | 4929 | rc); |
| 4930 | + | |
| 4931 | + return rc; | |
| 4925 | 4932 | } |
| 4926 | 4933 | |
| 4927 | 4934 | static netdev_features_t bnxt_fix_features(struct net_device *dev, |
| @@ -5212,13 +5219,27 @@ init_err: · 284 unmodified lines | ||
| 5212 | 5219 | static int bnxt_change_mac_addr(struct net_device *dev, void *p) |
| 5213 | 5220 | { |
| 5214 | 5221 | struct sockaddr *addr = p; |
| 5222 | + struct bnxt *bp = netdev_priv(dev); | |
| 5223 | + int rc = 0; | |
| 5215 | 5224 | |
| 5216 | 5225 | if (!is_valid_ether_addr(addr->sa_data)) |
| 5217 | 5226 | return -EADDRNOTAVAIL; |
| 5218 | 5227 | |
| 5228 | +#ifdef CONFIG_BNXT_SRIOV | |
| 5229 | + if (BNXT_VF(bp) && is_valid_ether_addr(bp->vf.mac_addr)) | |
| 5230 | + return -EADDRNOTAVAIL; | |
| 5231 | +#endif | |
| 5232 | + | |
| 5233 | + if (ether_addr_equal(addr->sa_data, dev->dev_addr)) | |
| 5234 | + return 0; | |
| 5235 | + | |
| 5219 | 5236 | memcpy(dev->dev_addr, addr->sa_data, dev->addr_len); |
| 5237 | + if (netif_running(dev)) { | |
| 5238 | + bnxt_close_nic(bp, false, false); | |
| 5239 | + rc = bnxt_open_nic(bp, false, false); | |
| 5240 | + } | |
| 5220 | 5241 | |
| 5221 | − return 0; | |
| 5242 | + return rc; | |
| 5222 | 5243 | } |
| 5223 | 5244 | |
| 5224 | 5245 | /* rtnl_lock held */ |
| @@ -5686,15 +5707,12 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · 461 unmodified lines | ||
| 5686 | 5707 | bnxt_set_tpa_flags(bp); |
| 5687 | 5708 | bnxt_set_ring_params(bp); |
| 5688 | 5709 | dflt_rings = netif_get_num_default_rss_queues(); |
| 5689 | − if (BNXT_PF(bp)) { | |
| 5690 | − memcpy(dev->dev_addr, bp->pf.mac_addr, ETH_ALEN); | |
| 5710 | + if (BNXT_PF(bp)) | |
| 5691 | 5711 | bp->pf.max_irqs = max_irqs; |
| 5692 | − } else { | |
| 5693 | 5712 | #if defined(CONFIG_BNXT_SRIOV) |
| 5694 | − memcpy(dev->dev_addr, bp->vf.mac_addr, ETH_ALEN); | |
| 5713 | + else | |
| 5695 | 5714 | bp->vf.max_irqs = max_irqs; |
| 5696 | 5715 | #endif |
| 5697 | − } | |
| 5698 | 5716 | bnxt_get_max_rings(bp, &max_rx_rings, &max_tx_rings); |
| 5699 | 5717 | bp->rx_nr_rings = min_t(int, dflt_rings, max_rx_rings); |
| 5700 | 5718 | bp->tx_nr_rings_per_tc = min_t(int, dflt_rings, max_tx_rings); |
| @@ -804,10 +804,9 @@ void bnxt_update_vf_mac(struct bnxt *bp) · 803 unmodified lines | ||
| 804 | 804 | if (!is_valid_ether_addr(resp->perm_mac_address)) |
| 805 | 805 | goto update_vf_mac_exit; |
| 806 | 806 | |
| 807 | − if (ether_addr_equal(resp->perm_mac_address, bp->vf.mac_addr)) | |
| 808 | − goto update_vf_mac_exit; | |
| 809 | − | |
| 810 | − memcpy(bp->vf.mac_addr, resp->perm_mac_address, ETH_ALEN); | |
| 807 | + if (!ether_addr_equal(resp->perm_mac_address, bp->vf.mac_addr)) | |
| 808 | + memcpy(bp->vf.mac_addr, resp->perm_mac_address, ETH_ALEN); | |
| 809 | + /* overwrite netdev dev_adr with admin VF MAC */ | |
| 811 | 810 | memcpy(bp->dev->dev_addr, bp->vf.mac_addr, ETH_ALEN); |
| 812 | 811 | update_vf_mac_exit: |
| 813 | 812 | mutex_unlock(&bp->hwrm_cmd_lock); |
| @@ -1682,6 +1682,8 @@ static void macb_init_hw(struct macb *bp) · 1681 unmodified lines | ||
| 1682 | 1682 | macb_set_hwaddr(bp); |
| 1683 | 1683 | |
| 1684 | 1684 | config = macb_mdc_clk_div(bp); |
| 1685 | + if (bp->phy_interface == PHY_INTERFACE_MODE_SGMII) | |
| 1686 | + config |= GEM_BIT(SGMIIEN) | GEM_BIT(PCSSEL); | |
| 1685 | 1687 | config |= MACB_BF(RBOF, NET_IP_ALIGN); /* Make eth data aligned */ |
| 1686 | 1688 | config |= MACB_BIT(PAE); /* PAuse Enable */ |
| 1687 | 1689 | config |= MACB_BIT(DRFCS); /* Discard Rx FCS */ |
| @@ -2416,6 +2418,8 @@ static int macb_init(struct platform_device *pdev) · 728 unmodified lines | ||
| 2416 | 2418 | /* Set MII management clock divider */ |
| 2417 | 2419 | val = macb_mdc_clk_div(bp); |
| 2418 | 2420 | val |= macb_dbw(bp); |
| 2421 | + if (bp->phy_interface == PHY_INTERFACE_MODE_SGMII) | |
| 2422 | + val |= GEM_BIT(SGMIIEN) | GEM_BIT(PCSSEL); | |
| 2419 | 2423 | macb_writel(bp, NCFGR, val); |
| 2420 | 2424 | |
| 2421 | 2425 | return 0; |
| @@ -215,12 +215,17 @@ · 214 unmodified lines | ||
| 215 | 215 | /* GEM specific NCFGR bitfields. */ |
| 216 | 216 | #define GEM_GBE_OFFSET 10 /* Gigabit mode enable */ |
| 217 | 217 | #define GEM_GBE_SIZE 1 |
| 218 | +#define GEM_PCSSEL_OFFSET 11 | |
| 219 | +#define GEM_PCSSEL_SIZE 1 | |
| 218 | 220 | #define GEM_CLK_OFFSET 18 /* MDC clock division */ |
| 219 | 221 | #define GEM_CLK_SIZE 3 |
| 220 | 222 | #define GEM_DBW_OFFSET 21 /* Data bus width */ |
| 221 | 223 | #define GEM_DBW_SIZE 2 |
| 222 | 224 | #define GEM_RXCOEN_OFFSET 24 |
| 223 | 225 | #define GEM_RXCOEN_SIZE 1 |
| 226 | +#define GEM_SGMIIEN_OFFSET 27 | |
| 227 | +#define GEM_SGMIIEN_SIZE 1 | |
| 228 | + | |
| 224 | 229 | |
| 225 | 230 | /* Constants for data bus width. */ |
| 226 | 231 | #define GEM_DBW32 0 /* 32 bit AMBA AHB data bus width */ |
| @@ -120,10 +120,9 @@ · 119 unmodified lines | ||
| 120 | 120 | * Calculated for SCLK of 700Mhz |
| 121 | 121 | * value written should be a 1/16th of what is expected |
| 122 | 122 | * |
| 123 | − * 1 tick per 0.05usec = value of 2.2 | |
| 124 | − * This 10% would be covered in CQ timer thresh value | |
| 123 | + * 1 tick per 0.025usec | |
| 125 | 124 | */ |
| 126 | −#define NICPF_CLK_PER_INT_TICK 2 | |
| 125 | +#define NICPF_CLK_PER_INT_TICK 1 | |
| 127 | 126 | |
| 128 | 127 | /* Time to wait before we decide that a SQ is stuck. |
| 129 | 128 | * |
| @@ -37,6 +37,7 @@ struct nicpf { · 36 unmodified lines | ||
| 37 | 37 | #define NIC_GET_BGX_FROM_VF_LMAC_MAP(map) ((map >> 4) & 0xF) |
| 38 | 38 | #define NIC_GET_LMAC_FROM_VF_LMAC_MAP(map) (map & 0xF) |
| 39 | 39 | u8 vf_lmac_map[MAX_LMAC]; |
| 40 | + u8 lmac_cnt; | |
| 40 | 41 | struct delayed_work dwork; |
| 41 | 42 | struct workqueue_struct *check_link; |
| 42 | 43 | u8 link[MAX_LMAC]; |
| @@ -279,6 +280,7 @@ static void nic_set_lmac_vf_mapping(struct nicpf *nic) · 236 unmodified lines | ||
| 279 | 280 | u64 lmac_credit; |
| 280 | 281 | |
| 281 | 282 | nic->num_vf_en = 0; |
| 283 | + nic->lmac_cnt = 0; | |
| 282 | 284 | |
| 283 | 285 | for (bgx = 0; bgx < NIC_MAX_BGX; bgx++) { |
| 284 | 286 | if (!(bgx_map & (1 << bgx))) |
| @@ -288,6 +290,7 @@ static void nic_set_lmac_vf_mapping(struct nicpf *nic) · 3 unmodified lines | ||
| 288 | 290 | nic->vf_lmac_map[next_bgx_lmac++] = |
| 289 | 291 | NIC_SET_VF_LMAC_MAP(bgx, lmac); |
| 290 | 292 | nic->num_vf_en += lmac_cnt; |
| 293 | + nic->lmac_cnt += lmac_cnt; | |
| 291 | 294 | |
| 292 | 295 | /* Program LMAC credits */ |
| 293 | 296 | lmac_credit = (1ull << 1); /* channel credit enable */ |
| @@ -715,6 +718,13 @@ static void nic_handle_mbx_intr(struct nicpf *nic, int vf) · 421 unmodified lines | ||
| 715 | 718 | case NIC_MBOX_MSG_CFG_DONE: |
| 716 | 719 | /* Last message of VF config msg sequence */ |
| 717 | 720 | nic->vf_enabled[vf] = true; |
| 721 | + if (vf >= nic->lmac_cnt) | |
| 722 | + goto unlock; | |
| 723 | + | |
| 724 | + bgx = NIC_GET_BGX_FROM_VF_LMAC_MAP(nic->vf_lmac_map[vf]); | |
| 725 | + lmac = NIC_GET_LMAC_FROM_VF_LMAC_MAP(nic->vf_lmac_map[vf]); | |
| 726 | + | |
| 727 | + bgx_lmac_rx_tx_enable(nic->node, bgx, lmac, true); | |
| 718 | 728 | goto unlock; |
| 719 | 729 | case NIC_MBOX_MSG_SHUTDOWN: |
| 720 | 730 | /* First msg in VF teardown sequence */ |
| @@ -722,6 +732,14 @@ static void nic_handle_mbx_intr(struct nicpf *nic, int vf) · 1 unmodified line | ||
| 722 | 732 | if (vf >= nic->num_vf_en) |
| 723 | 733 | nic->sqs_used[vf - nic->num_vf_en] = false; |
| 724 | 734 | nic->pqs_vf[vf] = 0; |
| 735 | + | |
| 736 | + if (vf >= nic->lmac_cnt) | |
| 737 | + break; | |
| 738 | + | |
| 739 | + bgx = NIC_GET_BGX_FROM_VF_LMAC_MAP(nic->vf_lmac_map[vf]); | |
| 740 | + lmac = NIC_GET_LMAC_FROM_VF_LMAC_MAP(nic->vf_lmac_map[vf]); | |
| 741 | + | |
| 742 | + bgx_lmac_rx_tx_enable(nic->node, bgx, lmac, false); | |
| 725 | 743 | break; |
| 726 | 744 | case NIC_MBOX_MSG_ALLOC_SQS: |
| 727 | 745 | nic_alloc_sqs(nic, &mbx.sqs_alloc); |
| @@ -940,7 +958,7 @@ static void nic_poll_for_link(struct work_struct *work) · 212 unmodified lines | ||
| 940 | 958 | |
| 941 | 959 | mbx.link_status.msg = NIC_MBOX_MSG_BGX_LINK_CHANGE; |
| 942 | 960 | |
| 943 | − for (vf = 0; vf < nic->num_vf_en; vf++) { | |
| 961 | + for (vf = 0; vf < nic->lmac_cnt; vf++) { | |
| 944 | 962 | /* Poll only if VF is UP */ |
| 945 | 963 | if (!nic->vf_enabled[vf]) |
11 more lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 197 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 400 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 572 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 126 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 307 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 207 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 196 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 331 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 226 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 153 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 238 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 121 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 459 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 179 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.