UAPI Changes: - drm/i915: Show per-engine default property values in sysfs By providing the default values configured into the kernel via sysfs, it is much more convenient for userspace to restore those sane defaults, or at least know what are considered good baseline. This is useful, for example, to cleanup after any failed userspace prior to commencing new jobs. Cross-subsystem Changes: - video/hdmi: Add Unpack only function for DRM infoframe - Includes pull request gvt-next-2020-05-12 Driver Changes: - Restore Cherryview back to full-ppgtt (Chris, Mika) - Document locking guidelines for i915 (Chris, Daniel, Joonas) - Fix GitLab #1746: Handle idling during i915_gem_evict_something busy loops (Chris) - Display WA #1105: Require linear fb stride to be multiple of 512 bytes on gen9/glk (Ville) - Add Wa_14010685332 for ICP/ICL (Matt R) - Restrict w/a 1607087056 for EHL/JSL (Swathi) - Fix interrupt handling for DP AUX transactions on Tigerlake (Imre) - Revert "drm/i915/tgl: Include ro parts of l3 to invalidate" (Mika) - Fix HDC pipeline flush hardware bit on Gen12 (Mika) - Flush L3 when flushing render on Gen12 (Mika) - Invalidate aux table entries forcibly between BB on Gen12 (Mika) - Add aux table invalidate for all engines on Gen12 (Mika) - Force pte cacheline to main memory Gen8+ (Mika) - Add and enable TGL+ SAGV support (Stanislav) - Implement vm_ops->access on i915 mmaps for GDB (Chris, Kristian) - Replace zero-length array with flexible-array (Gustavo) - Improve batch buffer pool effectiveness to mitigate soft-rc6 hit (Chris) - Remove wait priority boosting (Chris) - Keep driver module referenced when PMU is active (Chris) - Sanitize RPS interrupts upon resume (Chris) - Extend pcode read timeout to 20 ms (Chris) - Wait for ACT sent before enabling MST pipe (Ville) - Extend support to async relocations to SNB (Chris) - Remove CNL pre-prod workarounds (Ville) - Don't enable WaIncreaseLatencyIPCEnabled when IPC is disabled (Sultan) - Record the active CCID from before reset (Chris) - Mark concurrent submissions with a weak-dependency (Chris) - Peel dma-fence-chains for await to allow engine-to-engine sync (Lionel) - Prevent using semaphores to chain up to external fences (Chris) - Fix GLK watermark calculations (Ville) - Emit await(batch) before MI_BB_START (Chris) - Reset execlists registers before HWSP (Chris) - Drop no-semaphore boosting in favor of fast timeslicing (Chris) - Fix enabled infoframe states of lspcon (Gwan-gyeong) - Program DP SDPs on pipe updates (Gwan-gyeong) - Stop sending DP SDPs on ddi disable (Gwan-gyeong) - Store CS timestamp frequency in Hz (Ville) - Remove unused HAS_FWTABLE macro (Pascal) - Use batchbuffer chaining for relocations to save ring space (Chris) - Try different engines for relocs if MI ops not supported (Chris, Tvrtko) - Lazily acquire the device wakeref for freeing objects (Chris) - Streamline display code arithmetics around rounding etc. (Ville) - Use bw state for per crtc SAGV evaluation (Stanislav) - Track active_pipes in bw_state (Stanislav) - Nuke mode.vrefresh usage (Ville) - Warn if the FBC is still writing to stolen on removal (Chris) - Added new PCode commands prepping for QGV rescricting (Stansilav) - Stop holding onto the pinned_default_state (Chris) - Propagate error from completed fences (Chris) - Ignore submit-fences on the same timeline (Chris) - Pull waiting on an external dma-fence into its routine (Chris) - Replace the hardcoded I915_FENCE_TIMEOUT with Kconfig (Chris) - Mark up the racy read of execlists->context_tag (Chris) - Tidy up the return handling for completed dma-fences (Chris) - Introduce skl_plane_wm_level accessor (Stanislav) - Extract SKL SAGV checking (Stanislav) - Make active_pipes check skl specific (Stanislav) - Suspend tasklets before resume sanitization (Chris) - Remove redundant exec_fence (Chris) - Mark the addition of the initial-breadcrumb in the request (Chris) - Transfer old virtual breadcrumbs to irq_worker (Chris) - Read the DP SDPs from the video DIP (Gwan-gyeong) - Program DP SDPs with computed configs (Gwan-gyeong) - Add state readout for DP VSC and DP HDR Metadata Infoframe SDP (Gwan-gyeong) - Add compute routine for DP PSR VSC SDP (Gwan-gyeong) - Use new DP VSC SDP compute routine on PSR (Gwan-gyeong) - Restrict qgv points which don't have enough bandwidth. (Stanislav) - Nuke pointless div by 64bit (Ville) - Static checker code fixes (Nathan, Mika, Chris) - Add logging function for DP VSC SDP (Gwan-gyeong) - Include HDMI DRM infoframe, DP HDR metadata and DP VSC SDP in the crtc state dump (Gwan-gyeong) - Make timeslicing explicit engine property (Chris, Tvrtko) - Selftest and debugging improvements (Chris) - Align variable names with BSpec (Ville) - Tidy up gen8+ breadcrumb emission code (Chris) - Turn intel_digital_port_connected() in a vfunc (Ville) - Use stashed away hpd isr bits in intel_digital_port_connected() (Ville) - Extract i915_cs_timestamp_{ns_to_ticks,tick_to_ns}() (Ville)
Unified · Split
| @@ -329,6 +329,52 @@ for execution also include a list of all locations within buffers that · 328 unmodified lines | |||
| 329 | refer to GPU-addresses so that the kernel can edit the buffer correctly. | 329 | refer to GPU-addresses so that the kernel can edit the buffer correctly. |
| 330 | This process is dubbed relocation. | 330 | This process is dubbed relocation. |
| 331 | 331 | ||
| 332 | Locking Guidelines | ||
| 333 | ------------------ | ||
| 334 | |||
| 335 | .. note:: | ||
| 336 | This is a description of how the locking should be after | ||
| 337 | refactoring is done. Does not necessarily reflect what the locking | ||
| 338 | looks like while WIP. | ||
| 339 | |||
| 340 | #. All locking rules and interface contracts with cross-driver interfaces | ||
| 341 | (dma-buf, dma_fence) need to be followed. | ||
| 342 | |||
| 343 | #. No struct_mutex anywhere in the code | ||
| 344 | |||
| 345 | #. dma_resv will be the outermost lock (when needed) and ww_acquire_ctx | ||
| 346 | is to be hoisted at highest level and passed down within i915_gem_ctx | ||
| 347 | in the call chain | ||
| 348 | |||
| 349 | #. While holding lru/memory manager (buddy, drm_mm, whatever) locks | ||
| 350 | system memory allocations are not allowed | ||
| 351 | |||
| 352 | * Enforce this by priming lockdep (with fs_reclaim). If we | ||
| 353 | allocate memory while holding these looks we get a rehash | ||
| 354 | of the shrinker vs. struct_mutex saga, and that would be | ||
| 355 | real bad. | ||
| 356 | |||
| 357 | #. Do not nest different lru/memory manager locks within each other. | ||
| 358 | Take them in turn to update memory allocations, relying on the object’s | ||
| 359 | dma_resv ww_mutex to serialize against other operations. | ||
| 360 | |||
| 361 | #. The suggestion for lru/memory managers locks is that they are small | ||
| 362 | enough to be spinlocks. | ||
| 363 | |||
| 364 | #. All features need to come with exhaustive kernel selftests and/or | ||
| 365 | IGT tests when appropriate | ||
| 366 | |||
| 367 | #. All LMEM uAPI paths need to be fully restartable (_interruptible() | ||
| 368 | for all locks/waits/sleeps) | ||
| 369 | |||
| 370 | * Error handling validation through signal injection. | ||
| 371 | Still the best strategy we have for validating GEM uAPI | ||
| 372 | corner cases. | ||
| 373 | Must be excessively used in the IGT, and we need to check | ||
| 374 | that we really have full path coverage of all error cases. | ||
| 375 | |||
| 376 | * -EDEADLK handling with ww_mutex | ||
| 377 | |||
| 332 | GEM BO Management Implementation Details | 378 | GEM BO Management Implementation Details |
| 333 | ---------------------------------------- | 379 | ---------------------------------------- |
| 334 | 380 | ||
| @@ -1629,3 +1629,177 @@ int drm_dp_set_phy_test_pattern(struct drm_dp_aux *aux, · 1628 unmodified lines | |||
| 1629 | return 0; | 1629 | return 0; |
| 1630 | } | 1630 | } |
| 1631 | EXPORT_SYMBOL(drm_dp_set_phy_test_pattern); | 1631 | EXPORT_SYMBOL(drm_dp_set_phy_test_pattern); |
| 1632 | |||
| 1633 | static const char *dp_pixelformat_get_name(enum dp_pixelformat pixelformat) | ||
| 1634 | { | ||
| 1635 | if (pixelformat < 0 || pixelformat > DP_PIXELFORMAT_RESERVED) | ||
| 1636 | return "Invalid"; | ||
| 1637 | |||
| 1638 | switch (pixelformat) { | ||
| 1639 | case DP_PIXELFORMAT_RGB: | ||
| 1640 | return "RGB"; | ||
| 1641 | case DP_PIXELFORMAT_YUV444: | ||
| 1642 | return "YUV444"; | ||
| 1643 | case DP_PIXELFORMAT_YUV422: | ||
| 1644 | return "YUV422"; | ||
| 1645 | case DP_PIXELFORMAT_YUV420: | ||
| 1646 | return "YUV420"; | ||
| 1647 | case DP_PIXELFORMAT_Y_ONLY: | ||
| 1648 | return "Y_ONLY"; | ||
| 1649 | case DP_PIXELFORMAT_RAW: | ||
| 1650 | return "RAW"; | ||
| 1651 | default: | ||
| 1652 | return "Reserved"; | ||
| 1653 | } | ||
| 1654 | } | ||
| 1655 | |||
| 1656 | static const char *dp_colorimetry_get_name(enum dp_pixelformat pixelformat, | ||
| 1657 | enum dp_colorimetry colorimetry) | ||
| 1658 | { | ||
| 1659 | if (pixelformat < 0 || pixelformat > DP_PIXELFORMAT_RESERVED) | ||
| 1660 | return "Invalid"; | ||
| 1661 | |||
| 1662 | switch (colorimetry) { | ||
| 1663 | case DP_COLORIMETRY_DEFAULT: | ||
| 1664 | switch (pixelformat) { | ||
| 1665 | case DP_PIXELFORMAT_RGB: | ||
| 1666 | return "sRGB"; | ||
| 1667 | case DP_PIXELFORMAT_YUV444: | ||
| 1668 | case DP_PIXELFORMAT_YUV422: | ||
| 1669 | case DP_PIXELFORMAT_YUV420: | ||
| 1670 | return "BT.601"; | ||
| 1671 | case DP_PIXELFORMAT_Y_ONLY: | ||
| 1672 | return "DICOM PS3.14"; | ||
| 1673 | case DP_PIXELFORMAT_RAW: | ||
| 1674 | return "Custom Color Profile"; | ||
| 1675 | default: | ||
| 1676 | return "Reserved"; | ||
| 1677 | } | ||
| 1678 | case DP_COLORIMETRY_RGB_WIDE_FIXED: /* and DP_COLORIMETRY_BT709_YCC */ | ||
| 1679 | switch (pixelformat) { | ||
| 1680 | case DP_PIXELFORMAT_RGB: | ||
| 1681 | return "Wide Fixed"; | ||
| 1682 | case DP_PIXELFORMAT_YUV444: | ||
| 1683 | case DP_PIXELFORMAT_YUV422: | ||
| 1684 | case DP_PIXELFORMAT_YUV420: | ||
| 1685 | return "BT.709"; | ||
| 1686 | default: | ||
| 1687 | return "Reserved"; | ||
| 1688 | } | ||
| 1689 | case DP_COLORIMETRY_RGB_WIDE_FLOAT: /* and DP_COLORIMETRY_XVYCC_601 */ | ||
| 1690 | switch (pixelformat) { | ||
| 1691 | case DP_PIXELFORMAT_RGB: | ||
| 1692 | return "Wide Float"; | ||
| 1693 | case DP_PIXELFORMAT_YUV444: | ||
| 1694 | case DP_PIXELFORMAT_YUV422: | ||
| 1695 | case DP_PIXELFORMAT_YUV420: | ||
| 1696 | return "xvYCC 601"; | ||
| 1697 | default: | ||
| 1698 | return "Reserved"; | ||
| 1699 | } | ||
| 1700 | case DP_COLORIMETRY_OPRGB: /* and DP_COLORIMETRY_XVYCC_709 */ | ||
| 1701 | switch (pixelformat) { | ||
| 1702 | case DP_PIXELFORMAT_RGB: | ||
| 1703 | return "OpRGB"; | ||
| 1704 | case DP_PIXELFORMAT_YUV444: | ||
| 1705 | case DP_PIXELFORMAT_YUV422: | ||
| 1706 | case DP_PIXELFORMAT_YUV420: | ||
| 1707 | return "xvYCC 709"; | ||
| 1708 | default: | ||
| 1709 | return "Reserved"; | ||
| 1710 | } | ||
| 1711 | case DP_COLORIMETRY_DCI_P3_RGB: /* and DP_COLORIMETRY_SYCC_601 */ | ||
| 1712 | switch (pixelformat) { | ||
| 1713 | case DP_PIXELFORMAT_RGB: | ||
| 1714 | return "DCI-P3"; | ||
| 1715 | case DP_PIXELFORMAT_YUV444: | ||
| 1716 | case DP_PIXELFORMAT_YUV422: | ||
| 1717 | case DP_PIXELFORMAT_YUV420: | ||
| 1718 | return "sYCC 601"; | ||
| 1719 | default: | ||
| 1720 | return "Reserved"; | ||
| 1721 | } | ||
| 1722 | case DP_COLORIMETRY_RGB_CUSTOM: /* and DP_COLORIMETRY_OPYCC_601 */ | ||
| 1723 | switch (pixelformat) { | ||
| 1724 | case DP_PIXELFORMAT_RGB: | ||
| 1725 | return "Custom Profile"; | ||
| 1726 | case DP_PIXELFORMAT_YUV444: | ||
| 1727 | case DP_PIXELFORMAT_YUV422: | ||
| 1728 | case DP_PIXELFORMAT_YUV420: | ||
| 1729 | return "OpYCC 601"; | ||
| 1730 | default: | ||
| 1731 | return "Reserved"; | ||
| 1732 | } | ||
| 1733 | case DP_COLORIMETRY_BT2020_RGB: /* and DP_COLORIMETRY_BT2020_CYCC */ | ||
| 1734 | switch (pixelformat) { | ||
| 1735 | case DP_PIXELFORMAT_RGB: | ||
| 1736 | return "BT.2020 RGB"; | ||
| 1737 | case DP_PIXELFORMAT_YUV444: | ||
| 1738 | case DP_PIXELFORMAT_YUV422: | ||
| 1739 | case DP_PIXELFORMAT_YUV420: | ||
| 1740 | return "BT.2020 CYCC"; | ||
| 1741 | default: | ||
| 1742 | return "Reserved"; | ||
| 1743 | } | ||
| 1744 | case DP_COLORIMETRY_BT2020_YCC: | ||
| 1745 | switch (pixelformat) { | ||
| 1746 | case DP_PIXELFORMAT_YUV444: | ||
| 1747 | case DP_PIXELFORMAT_YUV422: | ||
| 1748 | case DP_PIXELFORMAT_YUV420: | ||
| 1749 | return "BT.2020 YCC"; | ||
| 1750 | default: | ||
| 1751 | return "Reserved"; | ||
| 1752 | } | ||
| 1753 | default: | ||
| 1754 | return "Invalid"; | ||
| 1755 | } | ||
| 1756 | } | ||
| 1757 | |||
| 1758 | static const char *dp_dynamic_range_get_name(enum dp_dynamic_range dynamic_range) | ||
| 1759 | { | ||
| 1760 | switch (dynamic_range) { | ||
| 1761 | case DP_DYNAMIC_RANGE_VESA: | ||
| 1762 | return "VESA range"; | ||
| 1763 | case DP_DYNAMIC_RANGE_CTA: | ||
| 1764 | return "CTA range"; | ||
| 1765 | default: | ||
| 1766 | return "Invalid"; | ||
| 1767 | } | ||
| 1768 | } | ||
| 1769 | |||
| 1770 | static const char *dp_content_type_get_name(enum dp_content_type content_type) | ||
| 1771 | { | ||
| 1772 | switch (content_type) { | ||
| 1773 | case DP_CONTENT_TYPE_NOT_DEFINED: | ||
| 1774 | return "Not defined"; | ||
| 1775 | case DP_CONTENT_TYPE_GRAPHICS: | ||
| 1776 | return "Graphics"; | ||
| 1777 | case DP_CONTENT_TYPE_PHOTO: | ||
| 1778 | return "Photo"; | ||
| 1779 | case DP_CONTENT_TYPE_VIDEO: | ||
| 1780 | return "Video"; | ||
| 1781 | case DP_CONTENT_TYPE_GAME: | ||
| 1782 | return "Game"; | ||
| 1783 | default: | ||
| 1784 | return "Reserved"; | ||
| 1785 | } | ||
| 1786 | } | ||
| 1787 | |||
| 1788 | void drm_dp_vsc_sdp_log(const char *level, struct device *dev, | ||
| 1789 | const struct drm_dp_vsc_sdp *vsc) | ||
| 1790 | { | ||
| 1791 | #define DP_SDP_LOG(fmt, ...) dev_printk(level, dev, fmt, ##__VA_ARGS__) | ||
| 1792 | DP_SDP_LOG("DP SDP: %s, revision %u, length %u\n", "VSC", | ||
| 1793 | vsc->revision, vsc->length); | ||
| 1794 | DP_SDP_LOG(" pixelformat: %s\n", | ||
| 1795 | dp_pixelformat_get_name(vsc->pixelformat)); | ||
| 1796 | DP_SDP_LOG(" colorimetry: %s\n", | ||
| 1797 | dp_colorimetry_get_name(vsc->pixelformat, vsc->colorimetry)); | ||
| 1798 | DP_SDP_LOG(" bpc: %u\n", vsc->bpc); | ||
| 1799 | DP_SDP_LOG(" dynamic range: %s\n", | ||
| 1800 | dp_dynamic_range_get_name(vsc->dynamic_range)); | ||
| 1801 | DP_SDP_LOG(" content type: %s\n", | ||
| 1802 | dp_content_type_get_name(vsc->content_type)); | ||
| 1803 | #undef DP_SDP_LOG | ||
| 1804 | } | ||
| 1805 | EXPORT_SYMBOL(drm_dp_vsc_sdp_log); | ||
| @@ -1,3 +1,15 @@ | |||
| 1 | config DRM_I915_FENCE_TIMEOUT | ||
| 2 | int "Timeout for unsignaled foreign fences (ms, jiffy granularity)" | ||
| 3 | default 10000 # milliseconds | ||
| 4 | help | ||
| 5 | When listening to a foreign fence, we install a supplementary timer | ||
| 6 | to ensure that we are always signaled and our userspace is able to | ||
| 7 | make forward progress. This value specifies the timeout used for an | ||
| 8 | unsignaled foreign fence. | ||
| 9 | |||
| 10 | May be 0 to disable the timeout, and rely on the foreign fence being | ||
| 11 | eventually signaled. | ||
| 12 | |||
| 1 | config DRM_I915_USERFAULT_AUTOSUSPEND | 13 | config DRM_I915_USERFAULT_AUTOSUSPEND |
| 2 | int "Runtime autosuspend delay for userspace GGTT mmaps (ms)" | 14 | int "Runtime autosuspend delay for userspace GGTT mmaps (ms)" |
| 3 | default 250 # milliseconds | 15 | default 250 # milliseconds |
| @@ -35,6 +35,7 @@ subdir-ccflags-y += -I$(srctree)/$(src) · 34 unmodified lines | |||
| 35 | 35 | ||
| 36 | # core driver code | 36 | # core driver code |
| 37 | i915-y += i915_drv.o \ | 37 | i915-y += i915_drv.o \ |
| 38 | i915_config.o \ | ||
| 38 | i915_irq.o \ | 39 | i915_irq.o \ |
| 39 | i915_getparam.o \ | 40 | i915_getparam.o \ |
| 40 | i915_params.o \ | 41 | i915_params.o \ |
| @@ -87,11 +88,11 @@ gt-y += \ · 46 unmodified lines | |||
| 87 | gt/intel_engine_cs.o \ | 88 | gt/intel_engine_cs.o \ |
| 88 | gt/intel_engine_heartbeat.o \ | 89 | gt/intel_engine_heartbeat.o \ |
| 89 | gt/intel_engine_pm.o \ | 90 | gt/intel_engine_pm.o \ |
| 90 | gt/intel_engine_pool.o \ | ||
| 91 | gt/intel_engine_user.o \ | 91 | gt/intel_engine_user.o \ |
| 92 | gt/intel_ggtt.o \ | 92 | gt/intel_ggtt.o \ |
| 93 | gt/intel_ggtt_fencing.o \ | 93 | gt/intel_ggtt_fencing.o \ |
| 94 | gt/intel_gt.o \ | 94 | gt/intel_gt.o \ |
| 95 | gt/intel_gt_buffer_pool.o \ | ||
| 95 | gt/intel_gt_clock_utils.o \ | 96 | gt/intel_gt_clock_utils.o \ |
| 96 | gt/intel_gt_irq.o \ | 97 | gt/intel_gt_irq.o \ |
| 97 | gt/intel_gt_pm.o \ | 98 | gt/intel_gt_pm.o \ |
| @@ -514,85 +514,67 @@ static void hsw_audio_codec_disable(struct intel_encoder *encoder, · 513 unmodified lines | |||
| 514 | mutex_unlock(&dev_priv->av_mutex); | 514 | mutex_unlock(&dev_priv->av_mutex); |
| 515 | } | 515 | } |
| 516 | 516 | ||
| 517 | /* Add a factor to take care of rounding and truncations */ | 517 | static unsigned int calc_hblank_early_prog(struct intel_encoder *encoder, |
| 518 | #define ROUNDING_FACTOR 10000 | 518 | const struct intel_crtc_state *crtc_state) |
| 519 | |||
| 520 | static unsigned int get_hblank_early_enable_config(struct intel_encoder *encoder, | ||
| 521 | const struct intel_crtc_state *crtc_state) | ||
| 522 | { | 519 | { |
| 523 | struct drm_i915_private *i915 = to_i915(encoder->base.dev); | 520 | struct drm_i915_private *i915 = to_i915(encoder->base.dev); |
| 524 | unsigned int link_clks_available, link_clks_required; | 521 | unsigned int link_clks_available, link_clks_required; |
| 525 | unsigned int tu_data, tu_line, link_clks_active; | 522 | unsigned int tu_data, tu_line, link_clks_active; |
| 526 | unsigned int hblank_rise, hblank_early_prog; | 523 | unsigned int h_active, h_total, hblank_delta, pixel_clk; |
| 527 | unsigned int h_active, h_total, hblank_delta, pixel_clk, v_total; | 524 | unsigned int fec_coeff, cdclk, vdsc_bpp; |
| 528 | unsigned int fec_coeff, refresh_rate, cdclk, vdsc_bpp; | 525 | unsigned int link_clk, lanes; |
| 526 | unsigned int hblank_rise; | ||
| 529 | 527 | ||
| 530 | h_active = crtc_state->hw.adjusted_mode.crtc_hdisplay; | 528 | h_active = crtc_state->hw.adjusted_mode.crtc_hdisplay; |
| 531 | h_total = crtc_state->hw.adjusted_mode.crtc_htotal; | 529 | h_total = crtc_state->hw.adjusted_mode.crtc_htotal; |
| 532 | v_total = crtc_state->hw.adjusted_mode.crtc_vtotal; | ||
| 533 | pixel_clk = crtc_state->hw.adjusted_mode.crtc_clock; | 530 | pixel_clk = crtc_state->hw.adjusted_mode.crtc_clock; |
| 534 | refresh_rate = crtc_state->hw.adjusted_mode.vrefresh; | ||
| 535 | vdsc_bpp = crtc_state->dsc.compressed_bpp; | 531 | vdsc_bpp = crtc_state->dsc.compressed_bpp; |
| 536 | cdclk = i915->cdclk.hw.cdclk; | 532 | cdclk = i915->cdclk.hw.cdclk; |
| 537 | /* fec= 0.972261, using rounding multiplier of 1000000 */ | 533 | /* fec= 0.972261, using rounding multiplier of 1000000 */ |
| 538 | fec_coeff = 972261; | 534 | fec_coeff = 972261; |
| 535 | link_clk = crtc_state->port_clock; | ||
| 536 | lanes = crtc_state->lane_count; | ||
| 539 | 537 | ||
| 540 | drm_dbg_kms(&i915->drm, "h_active = %u link_clk = %u :" | 538 | drm_dbg_kms(&i915->drm, "h_active = %u link_clk = %u :" |
| 541 | "lanes = %u vdsc_bpp = %u cdclk = %u\n", | 539 | "lanes = %u vdsc_bpp = %u cdclk = %u\n", |
| 542 | h_active, crtc_state->port_clock, crtc_state->lane_count, | 540 | h_active, link_clk, lanes, vdsc_bpp, cdclk); |
| 543 | vdsc_bpp, cdclk); | ||
| 544 | 541 | ||
| 545 | if (WARN_ON(!crtc_state->port_clock || !crtc_state->lane_count || | 542 | if (WARN_ON(!link_clk || !pixel_clk || !lanes || !vdsc_bpp || !cdclk)) |
| 546 | !crtc_state->dsc.compressed_bpp || !i915->cdclk.hw.cdclk)) | ||
| 547 | return 0; | 543 | return 0; |
| 548 | 544 | ||
| 549 | link_clks_available = ((((h_total - h_active) * | 545 | link_clks_available = (h_total - h_active) * link_clk / pixel_clk - 28; |
| 550 | ((crtc_state->port_clock * ROUNDING_FACTOR) / | 546 | link_clks_required = DIV_ROUND_UP(192000 * h_total, 1000 * pixel_clk) * (48 / lanes + 2); |
| 551 | pixel_clk)) / ROUNDING_FACTOR) - 28); | ||
| 552 | |||
| 553 | link_clks_required = DIV_ROUND_UP(192000, (refresh_rate * | ||
| 554 | v_total)) * ((48 / | ||
| 555 | crtc_state->lane_count) + 2); | ||
| 556 | 547 | ||
| 557 | if (link_clks_available > link_clks_required) | 548 | if (link_clks_available > link_clks_required) |
| 558 | hblank_delta = 32; | 549 | hblank_delta = 32; |
| 559 | else | 550 | else |
| 560 | hblank_delta = DIV_ROUND_UP(((((5 * ROUNDING_FACTOR) / | 551 | hblank_delta = DIV64_U64_ROUND_UP(mul_u32_u32(5 * (link_clk + cdclk), pixel_clk), |
| 561 | crtc_state->port_clock) + ((5 * | 552 | mul_u32_u32(link_clk, cdclk)); |
| 562 | ROUNDING_FACTOR) / | ||
| 563 | cdclk)) * pixel_clk), | ||
| 564 | ROUNDING_FACTOR); | ||
| 565 | |||
| 566 | tu_data = (pixel_clk * vdsc_bpp * 8) / ((crtc_state->port_clock * | ||
| 567 | crtc_state->lane_count * fec_coeff) / 1000000); | ||
| 568 | tu_line = (((h_active * crtc_state->port_clock * fec_coeff) / | ||
| 569 | 1000000) / (64 * pixel_clk)); | ||
| 570 | link_clks_active = (tu_line - 1) * 64 + tu_data; | ||
| 571 | 553 | ||
| 572 | hblank_rise = ((link_clks_active + 6 * DIV_ROUND_UP(link_clks_active, | 554 | tu_data = div64_u64(mul_u32_u32(pixel_clk * vdsc_bpp * 8, 1000000), |
| 573 | 250) + 4) * ((pixel_clk * ROUNDING_FACTOR) / | 555 | mul_u32_u32(link_clk * lanes, fec_coeff)); |
| 574 | crtc_state->port_clock)) / ROUNDING_FACTOR; | 556 | tu_line = div64_u64(h_active * mul_u32_u32(link_clk, fec_coeff), |
| 557 | mul_u32_u32(64 * pixel_clk, 1000000)); | ||
| 558 | link_clks_active = (tu_line - 1) * 64 + tu_data; | ||
| 575 | 559 | ||
| 576 | hblank_early_prog = h_active - hblank_rise + hblank_delta; | 560 | hblank_rise = (link_clks_active + 6 * DIV_ROUND_UP(link_clks_active, 250) + 4) * pixel_clk / link_clk; |
| 577 | 561 | ||
| 578 | return hblank_early_prog; | 562 | return h_active - hblank_rise + hblank_delta; |
| 579 | } | 563 | } |
| 580 | 564 | ||
| 581 | static unsigned int get_sample_room_req_config(const struct intel_crtc_state *crtc_state) | 565 | static unsigned int calc_samples_room(const struct intel_crtc_state *crtc_state) |
| 582 | { | 566 | { |
| 583 | unsigned int h_active, h_total, pixel_clk; | 567 | unsigned int h_active, h_total, pixel_clk; |
| 584 | unsigned int samples_room; | 568 | unsigned int link_clk, lanes; |
| 585 | 569 | ||
| 586 | h_active = crtc_state->hw.adjusted_mode.hdisplay; | 570 | h_active = crtc_state->hw.adjusted_mode.hdisplay; |
| 587 | h_total = crtc_state->hw.adjusted_mode.htotal; | 571 | h_total = crtc_state->hw.adjusted_mode.htotal; |
| 588 | pixel_clk = crtc_state->hw.adjusted_mode.clock; | 572 | pixel_clk = crtc_state->hw.adjusted_mode.clock; |
| 573 | link_clk = crtc_state->port_clock; | ||
| 574 | lanes = crtc_state->lane_count; | ||
| 589 | 575 | ||
| 590 | samples_room = ((((h_total - h_active) * ((crtc_state->port_clock * | 576 | return ((h_total - h_active) * link_clk - 12 * pixel_clk) / |
| 591 | ROUNDING_FACTOR) / pixel_clk)) / | 577 | (pixel_clk * (48 / lanes + 2)); |
| 592 | ROUNDING_FACTOR) - 12) / ((48 / | ||
| 593 | crtc_state->lane_count) + 2); | ||
| 594 | |||
| 595 | return samples_room; | ||
| 596 | } | 578 | } |
| 597 | 579 | ||
| 598 | static void enable_audio_dsc_wa(struct intel_encoder *encoder, | 580 | static void enable_audio_dsc_wa(struct intel_encoder *encoder, |
| @@ -618,8 +600,7 @@ static void enable_audio_dsc_wa(struct intel_encoder *encoder, · 19 unmodified lines | |||
| 618 | (crtc_state->hw.adjusted_mode.hdisplay >= 3840 && | 600 | (crtc_state->hw.adjusted_mode.hdisplay >= 3840 && |
| 619 | crtc_state->hw.adjusted_mode.vdisplay >= 2160)) { | 601 | crtc_state->hw.adjusted_mode.vdisplay >= 2160)) { |
| 620 | /* Get hblank early enable value required */ | 602 | /* Get hblank early enable value required */ |
| 621 | hblank_early_prog = get_hblank_early_enable_config(encoder, | 603 | hblank_early_prog = calc_hblank_early_prog(encoder, crtc_state); |
| 622 | crtc_state); | ||
| 623 | if (hblank_early_prog < 32) { | 604 | if (hblank_early_prog < 32) { |
| 624 | val &= ~HBLANK_START_COUNT_MASK(pipe); | 605 | val &= ~HBLANK_START_COUNT_MASK(pipe); |
| 625 | val |= HBLANK_START_COUNT(pipe, HBLANK_START_COUNT_32); | 606 | val |= HBLANK_START_COUNT(pipe, HBLANK_START_COUNT_32); |
| @@ -635,7 +616,7 @@ static void enable_audio_dsc_wa(struct intel_encoder *encoder, · 9 unmodified lines | |||
| 635 | } | 616 | } |
| 636 | 617 | ||
| 637 | /* Get samples room value required */ | 618 | /* Get samples room value required */ |
| 638 | samples_room = get_sample_room_req_config(crtc_state); | 619 | samples_room = calc_samples_room(crtc_state); |
| 639 | if (samples_room < 3) { | 620 | if (samples_room < 3) { |
| 640 | val &= ~NUMBER_SAMPLES_PER_LINE_MASK(pipe); | 621 | val &= ~NUMBER_SAMPLES_PER_LINE_MASK(pipe); |
| 641 | val |= NUMBER_SAMPLES_PER_LINE(pipe, samples_room); | 622 | val |= NUMBER_SAMPLES_PER_LINE(pipe, samples_room); |
| @@ -8,6 +8,9 @@ · 7 unmodified lines | |||
| 8 | #include "intel_bw.h" | 8 | #include "intel_bw.h" |
| 9 | #include "intel_display_types.h" | 9 | #include "intel_display_types.h" |
| 10 | #include "intel_sideband.h" | 10 | #include "intel_sideband.h" |
| 11 | #include "intel_atomic.h" | ||
| 12 | #include "intel_pm.h" | ||
| 13 | |||
| 11 | 14 | ||
| 12 | /* Parameters for Qclk Geyserville (QGV) */ | 15 | /* Parameters for Qclk Geyserville (QGV) */ |
| 13 | struct intel_qgv_point { | 16 | struct intel_qgv_point { |
| @@ -113,6 +116,26 @@ static int icl_pcode_read_qgv_point_info(struct drm_i915_private *dev_priv, · 99 unmodified lines | |||
| 113 | return 0; | 116 | return 0; |
| 114 | } | 117 | } |
| 115 | 118 | ||
| 119 | int icl_pcode_restrict_qgv_points(struct drm_i915_private *dev_priv, | ||
| 120 | u32 points_mask) | ||
| 121 | { | ||
| 122 | int ret; | ||
| 123 | |||
| 124 | /* bspec says to keep retrying for at least 1 ms */ | ||
| 125 | ret = skl_pcode_request(dev_priv, ICL_PCODE_SAGV_DE_MEM_SS_CONFIG, | ||
| 126 | points_mask, | ||
| 127 | ICL_PCODE_POINTS_RESTRICTED_MASK, | ||
| 128 | ICL_PCODE_POINTS_RESTRICTED, | ||
| 129 | 1); | ||
| 130 | |||
| 131 | if (ret < 0) { | ||
| 132 | drm_err(&dev_priv->drm, "Failed to disable qgv points (%d)\n", ret); | ||
| 133 | return ret; | ||
| 134 | } | ||
| 135 | |||
| 136 | return 0; | ||
| 137 | } | ||
| 138 | |||
| 116 | static int icl_get_qgv_points(struct drm_i915_private *dev_priv, | 139 | static int icl_get_qgv_points(struct drm_i915_private *dev_priv, |
| 117 | struct intel_qgv_info *qi) | 140 | struct intel_qgv_info *qi) |
| 118 | { | 141 | { |
| @@ -240,6 +263,16 @@ static int icl_get_bw_info(struct drm_i915_private *dev_priv, const struct intel · 121 unmodified lines | |||
| 240 | break; | 263 | break; |
| 241 | } | 264 | } |
| 242 | 265 | ||
| 266 | /* | ||
| 267 | * In case if SAGV is disabled in BIOS, we always get 1 | ||
| 268 | * SAGV point, but we can't send PCode commands to restrict it | ||
| 269 | * as it will fail and pointless anyway. | ||
| 270 | */ | ||
| 271 | if (qi.num_points == 1) | ||
| 272 | dev_priv->sagv_status = I915_SAGV_NOT_CONTROLLED; | ||
| 273 | else | ||
| 274 | dev_priv->sagv_status = I915_SAGV_ENABLED; | ||
| 275 | |||
| 243 | return 0; | 276 | return 0; |
| 244 | } | 277 | } |
| 245 | 278 | ||
| @@ -248,6 +281,11 @@ static unsigned int icl_max_bw(struct drm_i915_private *dev_priv, · 2 unmodified lines | |||
| 248 | { | 281 | { |
| 249 | int i; | 282 | int i; |
| 250 | 283 | ||
| 284 | /* | ||
| 285 | * Let's return max bw for 0 planes | ||
| 286 | */ | ||
| 287 | num_planes = max(1, num_planes); | ||
| 288 | |||
| 251 | for (i = 0; i < ARRAY_SIZE(dev_priv->max_bw); i++) { | 289 | for (i = 0; i < ARRAY_SIZE(dev_priv->max_bw); i++) { |
| 252 | const struct intel_bw_info *bi = | 290 | const struct intel_bw_info *bi = |
| 253 | &dev_priv->max_bw[i]; | 291 | &dev_priv->max_bw[i]; |
| @@ -277,34 +315,6 @@ void intel_bw_init_hw(struct drm_i915_private *dev_priv) · 23 unmodified lines | |||
| 277 | icl_get_bw_info(dev_priv, &icl_sa_info); | 315 | icl_get_bw_info(dev_priv, &icl_sa_info); |
| 278 | } | 316 | } |
| 279 | 317 | ||
| 280 | static unsigned int intel_max_data_rate(struct drm_i915_private *dev_priv, | ||
| 281 | int num_planes) | ||
| 282 | { | ||
| 283 | if (INTEL_GEN(dev_priv) >= 11) { | ||
| 284 | /* | ||
| 285 | * Any bw group has same amount of QGV points | ||
| 286 | */ | ||
| 287 | const struct intel_bw_info *bi = | ||
| 288 | &dev_priv->max_bw[0]; | ||
| 289 | unsigned int min_bw = UINT_MAX; | ||
| 290 | int i; | ||
| 291 | |||
| 292 | /* | ||
| 293 | * FIXME with SAGV disabled maybe we can assume | ||
| 294 | * point 1 will always be used? Seems to match | ||
| 295 | * the behaviour observed in the wild. | ||
| 296 | */ | ||
| 297 | for (i = 0; i < bi->num_qgv_points; i++) { | ||
| 298 | unsigned int bw = icl_max_bw(dev_priv, num_planes, i); | ||
| 299 | |||
| 300 | min_bw = min(bw, min_bw); | ||
| 301 | } | ||
| 302 | return min_bw; | ||
| 303 | } else { | ||
| 304 | return UINT_MAX; | ||
| 305 | } | ||
| 306 | } | ||
| 307 | |||
| 308 | static unsigned int intel_bw_crtc_num_active_planes(const struct intel_crtc_state *crtc_state) | 318 | static unsigned int intel_bw_crtc_num_active_planes(const struct intel_crtc_state *crtc_state) |
| 309 | { | 319 | { |
| 310 | /* | 320 | /* |
| @@ -414,11 +424,16 @@ int intel_bw_atomic_check(struct intel_atomic_state *state) · 103 unmodified lines | |||
| 414 | { | 424 | { |
| 415 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); | 425 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); |
| 416 | struct intel_crtc_state *new_crtc_state, *old_crtc_state; | 426 | struct intel_crtc_state *new_crtc_state, *old_crtc_state; |
| 417 | struct intel_bw_state *bw_state = NULL; | 427 | struct intel_bw_state *new_bw_state = NULL; |
| 418 | unsigned int data_rate, max_data_rate; | 428 | const struct intel_bw_state *old_bw_state = NULL; |
| 429 | unsigned int data_rate; | ||
| 419 | unsigned int num_active_planes; | 430 | unsigned int num_active_planes; |
| 420 | struct intel_crtc *crtc; | 431 | struct intel_crtc *crtc; |
| 421 | int i, ret; | 432 | int i, ret; |
| 433 | u32 allowed_points = 0; | ||
| 434 | unsigned int max_bw_point = 0, max_bw = 0; | ||
| 435 | unsigned int num_qgv_points = dev_priv->max_bw[0].num_qgv_points; | ||
| 436 | u32 mask = (1 << num_qgv_points) - 1; | ||
| 422 | 437 | ||
| 423 | /* FIXME earlier gens need some checks too */ | 438 | /* FIXME earlier gens need some checks too */ |
| 424 | if (INTEL_GEN(dev_priv) < 11) | 439 | if (INTEL_GEN(dev_priv) < 11) |
| @@ -443,41 +458,93 @@ int intel_bw_atomic_check(struct intel_atomic_state *state) · 18 unmodified lines | |||
| 443 | old_active_planes == new_active_planes) | 458 | old_active_planes == new_active_planes) |
| 444 | continue; | 459 | continue; |
| 445 | 460 | ||
| 446 | bw_state = intel_atomic_get_bw_state(state); | 461 | new_bw_state = intel_atomic_get_bw_state(state); |
| 447 | if (IS_ERR(bw_state)) | 462 | if (IS_ERR(new_bw_state)) |
| 448 | return PTR_ERR(bw_state); | 463 | return PTR_ERR(new_bw_state); |
| 449 | 464 | ||
| 450 | bw_state->data_rate[crtc->pipe] = new_data_rate; | 465 | new_bw_state->data_rate[crtc->pipe] = new_data_rate; |
| 451 | bw_state->num_active_planes[crtc->pipe] = new_active_planes; | 466 | new_bw_state->num_active_planes[crtc->pipe] = new_active_planes; |
| 452 | 467 | ||
| 453 | drm_dbg_kms(&dev_priv->drm, | 468 | drm_dbg_kms(&dev_priv->drm, |
| 454 | "pipe %c data rate %u num active planes %u\n", | 469 | "pipe %c data rate %u num active planes %u\n", |
| 455 | pipe_name(crtc->pipe), | 470 | pipe_name(crtc->pipe), |
| 456 | bw_state->data_rate[crtc->pipe], | 471 | new_bw_state->data_rate[crtc->pipe], |
| 457 | bw_state->num_active_planes[crtc->pipe]); | 472 | new_bw_state->num_active_planes[crtc->pipe]); |
| 458 | } | 473 | } |
| 459 | 474 | ||
| 460 | if (!bw_state) | 475 | if (!new_bw_state) |
| 461 | return 0; | 476 | return 0; |
| 462 | 477 | ||
| 463 | ret = intel_atomic_lock_global_state(&bw_state->base); | 478 | ret = intel_atomic_lock_global_state(&new_bw_state->base); |
| 464 | if (ret) | 479 | if (ret) |
| 465 | return ret; | 480 | return ret; |
| 466 | 481 | ||
| 467 | data_rate = intel_bw_data_rate(dev_priv, bw_state); | 482 | data_rate = intel_bw_data_rate(dev_priv, new_bw_state); |
| 468 | num_active_planes = intel_bw_num_active_planes(dev_priv, bw_state); | 483 | data_rate = DIV_ROUND_UP(data_rate, 1000); |
| 469 | 484 | ||
| 470 | max_data_rate = intel_max_data_rate(dev_priv, num_active_planes); | 485 | num_active_planes = intel_bw_num_active_planes(dev_priv, new_bw_state); |
| 471 | 486 | ||
| 472 | data_rate = DIV_ROUND_UP(data_rate, 1000); | 487 | for (i = 0; i < num_qgv_points; i++) { |
| 488 | unsigned int max_data_rate; | ||
| 473 | 489 | ||
| 474 | if (data_rate > max_data_rate) { | 490 | max_data_rate = icl_max_bw(dev_priv, num_active_planes, i); |
| 475 | drm_dbg_kms(&dev_priv->drm, | 491 | /* |
| 476 | "Bandwidth %u MB/s exceeds max available %d MB/s (%d active planes)\n", | 492 | * We need to know which qgv point gives us |
| 477 | data_rate, max_data_rate, num_active_planes); | 493 | * maximum bandwidth in order to disable SAGV |
| 494 | * if we find that we exceed SAGV block time | ||
| 495 | * with watermarks. By that moment we already | ||
| 496 | * have those, as it is calculated earlier in | ||
| 497 | * intel_atomic_check, | ||
| 498 | */ | ||
| 499 | if (max_data_rate > max_bw) { | ||
| 500 | max_bw_point = i; | ||
| 501 | max_bw = max_data_rate; | ||
| 502 | } | ||
| 503 | if (max_data_rate >= data_rate) | ||
| 504 | allowed_points |= BIT(i); | ||
| 505 | drm_dbg_kms(&dev_priv->drm, "QGV point %d: max bw %d required %d\n", | ||
| 506 | i, max_data_rate, data_rate); | ||
| 507 | } | ||
| 508 | |||
| 509 | /* | ||
| 510 | * BSpec states that we always should have at least one allowed point | ||
| 511 | * left, so if we couldn't - simply reject the configuration for obvious | ||
| 512 | * reasons. | ||
| 513 | */ | ||
| 514 | if (allowed_points == 0) { | ||
| 515 | drm_dbg_kms(&dev_priv->drm, "No QGV points provide sufficient memory" | ||
| 516 | " bandwidth %d for display configuration(%d active planes).\n", | ||
| 517 | data_rate, num_active_planes); | ||
| 478 | return -EINVAL; | 518 | return -EINVAL; |
| 479 | } | 519 | } |
| 480 | 520 | ||
| 521 | /* | ||
| 522 | * Leave only single point with highest bandwidth, if | ||
| 523 | * we can't enable SAGV due to the increased memory latency it may | ||
| 524 | * cause. | ||
| 525 | */ | ||
| 526 | if (!intel_can_enable_sagv(dev_priv, new_bw_state)) { | ||
| 527 | allowed_points = BIT(max_bw_point); | ||
| 528 | drm_dbg_kms(&dev_priv->drm, "No SAGV, using single QGV point %d\n", | ||
| 529 | max_bw_point); | ||
| 530 | } | ||
| 531 | /* | ||
| 532 | * We store the ones which need to be masked as that is what PCode | ||
| 533 | * actually accepts as a parameter. | ||
| 534 | */ | ||
| 535 | new_bw_state->qgv_points_mask = ~allowed_points & mask; | ||
| 536 | |||
| 537 | old_bw_state = intel_atomic_get_old_bw_state(state); | ||
| 538 | /* | ||
| 539 | * If the actual mask had changed we need to make sure that | ||
| 540 | * the commits are serialized(in case this is a nomodeset, nonblocking) | ||
| 541 | */ | ||
| 542 | if (new_bw_state->qgv_points_mask != old_bw_state->qgv_points_mask) { | ||
| 543 | ret = intel_atomic_serialize_global_state(&new_bw_state->base); | ||
| 544 | if (ret) | ||
| 545 | return ret; | ||
| 546 | } | ||
| 547 | |||
| 481 | return 0; | 548 | return 0; |
| 482 | } | 549 | } |
| 483 | 550 | ||
| @@ -18,8 +18,24 @@ struct intel_crtc_state; · 17 unmodified lines | |||
| 18 | struct intel_bw_state { | 18 | struct intel_bw_state { |
| 19 | struct intel_global_state base; | 19 | struct intel_global_state base; |
| 20 | 20 | ||
| 21 | /* | ||
| 22 | * Contains a bit mask, used to determine, whether correspondent | ||
| 23 | * pipe allows SAGV or not. | ||
| 24 | */ | ||
| 25 | u8 pipe_sagv_reject; | ||
| 26 | |||
| 27 | /* | ||
| 28 | * Current QGV points mask, which restricts | ||
| 29 | * some particular SAGV states, not to confuse | ||
| 30 | * with pipe_sagv_mask. | ||
| 31 | */ | ||
| 32 | u8 qgv_points_mask; | ||
| 33 | |||
| 21 | unsigned int data_rate[I915_MAX_PIPES]; | 34 | unsigned int data_rate[I915_MAX_PIPES]; |
| 22 | u8 num_active_planes[I915_MAX_PIPES]; | 35 | u8 num_active_planes[I915_MAX_PIPES]; |
| 36 | |||
| 37 | /* bitmask of active pipes */ | ||
| 38 | u8 active_pipes; | ||
| 23 | }; | 39 | }; |
| 24 | 40 | ||
| 25 | #define to_intel_bw_state(x) container_of((x), struct intel_bw_state, base) | 41 | #define to_intel_bw_state(x) container_of((x), struct intel_bw_state, base) |
| @@ -38,5 +54,7 @@ int intel_bw_init(struct drm_i915_private *dev_priv); · 12 unmodified lines | |||
| 38 | int intel_bw_atomic_check(struct intel_atomic_state *state); | 54 | int intel_bw_atomic_check(struct intel_atomic_state *state); |
| 39 | void intel_bw_crtc_update(struct intel_bw_state *bw_state, | 55 | void intel_bw_crtc_update(struct intel_bw_state *bw_state, |
| 40 | const struct intel_crtc_state *crtc_state); | 56 | const struct intel_crtc_state *crtc_state); |
| 57 | int icl_pcode_restrict_qgv_points(struct drm_i915_private *dev_priv, | ||
| 58 | u32 points_mask); | ||
| 41 | 59 | ||
| 42 | #endif /* __INTEL_BW_H__ */ | 60 | #endif /* __INTEL_BW_H__ */ |
| @@ -3472,6 +3472,8 @@ static void intel_ddi_post_disable_dp(struct intel_atomic_state *state, · 3471 unmodified lines | |||
| 3472 | INTEL_OUTPUT_DP_MST); | 3472 | INTEL_OUTPUT_DP_MST); |
| 3473 | enum phy phy = intel_port_to_phy(dev_priv, encoder->port); | 3473 | enum phy phy = intel_port_to_phy(dev_priv, encoder->port); |
| 3474 | 3474 | ||
| 3475 | intel_dp_set_infoframes(encoder, false, old_crtc_state, old_conn_state); | ||
| 3476 | |||
| 3475 | /* | 3477 | /* |
| 3476 | * Power down sink before disabling the port, otherwise we end | 3478 | * Power down sink before disabling the port, otherwise we end |
| 3477 | * up getting interrupts from the sink on detecting link loss. | 3479 | * up getting interrupts from the sink on detecting link loss. |
| @@ -3680,9 +3682,8 @@ static void intel_enable_ddi_dp(struct intel_atomic_state *state, · 202 unmodified lines | |||
| 3680 | intel_dp_stop_link_train(intel_dp); | 3682 | intel_dp_stop_link_train(intel_dp); |
| 3681 | 3683 | ||
| 3682 | intel_edp_backlight_on(crtc_state, conn_state); | 3684 | intel_edp_backlight_on(crtc_state, conn_state); |
| 3683 | intel_psr_enable(intel_dp, crtc_state); | 3685 | intel_psr_enable(intel_dp, crtc_state, conn_state); |
| 3684 | intel_dp_vsc_enable(intel_dp, crtc_state, conn_state); | 3686 | intel_dp_set_infoframes(encoder, true, crtc_state, conn_state); |
| 3685 | intel_dp_hdr_metadata_enable(intel_dp, crtc_state, conn_state); | ||
| 3686 | intel_edp_drrs_enable(intel_dp, crtc_state); | 3687 | intel_edp_drrs_enable(intel_dp, crtc_state); |
| 3687 | 3688 | ||
| 3688 | if (crtc_state->has_audio) | 3689 | if (crtc_state->has_audio) |
| @@ -3864,7 +3865,8 @@ static void intel_ddi_update_pipe_dp(struct intel_atomic_state *state, · 175 unmodified lines | |||
| 3864 | 3865 | ||
| 3865 | intel_ddi_set_dp_msa(crtc_state, conn_state); | 3866 | intel_ddi_set_dp_msa(crtc_state, conn_state); |
| 3866 | 3867 | ||
| 3867 | intel_psr_update(intel_dp, crtc_state); | 3868 | intel_psr_update(intel_dp, crtc_state, conn_state); |
| 3869 | intel_dp_set_infoframes(encoder, true, crtc_state, conn_state); | ||
| 3868 | intel_edp_drrs_enable(intel_dp, crtc_state); | 3870 | intel_edp_drrs_enable(intel_dp, crtc_state); |
| 3869 | 3871 | ||
| 3870 | intel_panel_update_backlight(state, encoder, crtc_state, conn_state); | 3872 | intel_panel_update_backlight(state, encoder, crtc_state, conn_state); |
| @@ -4235,6 +4237,9 @@ void intel_ddi_get_config(struct intel_encoder *encoder, · 364 unmodified lines | |||
| 4235 | pipe_config->fec_enable); | 4237 | pipe_config->fec_enable); |
| 4236 | } | 4238 | } |
| 4237 | 4239 | ||
| 4240 | pipe_config->infoframes.enable |= | ||
| 4241 | intel_hdmi_infoframes_enabled(encoder, pipe_config); | ||
| 4242 | |||
| 4238 | break; | 4243 | break; |
| 4239 | case TRANS_DDI_MODE_SELECT_DP_MST: | 4244 | case TRANS_DDI_MODE_SELECT_DP_MST: |
| 4240 | pipe_config->output_types |= BIT(INTEL_OUTPUT_DP_MST); | 4245 | pipe_config->output_types |= BIT(INTEL_OUTPUT_DP_MST); |
| @@ -4246,6 +4251,9 @@ void intel_ddi_get_config(struct intel_encoder *encoder, · 5 unmodified lines | |||
| 4246 | REG_FIELD_GET(TRANS_DDI_MST_TRANSPORT_SELECT_MASK, temp); | 4251 | REG_FIELD_GET(TRANS_DDI_MST_TRANSPORT_SELECT_MASK, temp); |
| 4247 | 4252 | ||
| 4248 | intel_dp_get_m_n(intel_crtc, pipe_config); | 4253 | intel_dp_get_m_n(intel_crtc, pipe_config); |
| 4254 | |||
| 4255 | pipe_config->infoframes.enable |= | ||
| 4256 | intel_hdmi_infoframes_enabled(encoder, pipe_config); | ||
| 4249 | break; | 4257 | break; |
| 4250 | default: | 4258 | default: |
| 4251 | break; | 4259 | break; |
| @@ -4300,6 +4308,9 @@ void intel_ddi_get_config(struct intel_encoder *encoder, · 48 unmodified lines | |||
| 4300 | 4308 | ||
| 4301 | if (INTEL_GEN(dev_priv) >= 8) | 4309 | if (INTEL_GEN(dev_priv) >= 8) |
| 4302 | bdw_get_trans_port_sync_config(pipe_config); | 4310 | bdw_get_trans_port_sync_config(pipe_config); |
| 4311 | |||
| 4312 | intel_read_dp_sdp(encoder, pipe_config, HDMI_PACKET_TYPE_GAMUT_METADATA); | ||
| 4313 | intel_read_dp_sdp(encoder, pipe_config, DP_SDP_VSC); | ||
| 4303 | } | 4314 | } |
| 4304 | 4315 | ||
| 4305 | static enum intel_output_type | 4316 | static enum intel_output_type |
| @@ -4689,6 +4700,30 @@ intel_ddi_hotplug(struct intel_encoder *encoder, · 383 unmodified lines | |||
| 4689 | return state; | 4700 | return state; |
| 4690 | } | 4701 | } |
| 4691 | 4702 | ||
| 4703 | static bool lpt_digital_port_connected(struct intel_encoder *encoder) | ||
| 4704 | { | ||
| 4705 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 4706 | u32 bit = dev_priv->hotplug.pch_hpd[encoder->hpd_pin]; | ||
| 4707 | |||
| 4708 | return intel_de_read(dev_priv, SDEISR) & bit; | ||
| 4709 | } | ||
| 4710 | |||
| 4711 | static bool hsw_digital_port_connected(struct intel_encoder *encoder) | ||
| 4712 | { | ||
| 4713 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 4714 | u32 bit = dev_priv->hotplug.hpd[encoder->hpd_pin]; | ||
| 4715 | |||
| 4716 | return intel_de_read(dev_priv, DEISR) & bit; | ||
| 4717 | } | ||
| 4718 | |||
| 4719 | static bool bdw_digital_port_connected(struct intel_encoder *encoder) | ||
| 4720 | { | ||
| 4721 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 4722 | u32 bit = dev_priv->hotplug.hpd[encoder->hpd_pin]; | ||
| 4723 | |||
| 4724 | return intel_de_read(dev_priv, GEN8_DE_PORT_ISR) & bit; | ||
| 4725 | } | ||
| 4726 | |||
| 4692 | static struct intel_connector * | 4727 | static struct intel_connector * |
| 4693 | intel_ddi_init_hdmi_connector(struct intel_digital_port *intel_dig_port) | 4728 | intel_ddi_init_hdmi_connector(struct intel_digital_port *intel_dig_port) |
| 4694 | { | 4729 | { |
| @@ -4885,6 +4920,23 @@ void intel_ddi_init(struct drm_i915_private *dev_priv, enum port port) · 190 unmodified lines | |||
| 4885 | port_name(port)); | 4920 | port_name(port)); |
| 4886 | } | 4921 | } |
| 4887 | 4922 | ||
| 4923 | if (INTEL_GEN(dev_priv) >= 11) { | ||
| 4924 | if (intel_phy_is_tc(dev_priv, phy)) | ||
| 4925 | intel_dig_port->connected = intel_tc_port_connected; | ||
| 4926 | else | ||
| 4927 | intel_dig_port->connected = lpt_digital_port_connected; | ||
| 4928 | } else if (INTEL_GEN(dev_priv) >= 8) { | ||
| 4929 | if (port == PORT_A || IS_GEN9_LP(dev_priv)) | ||
| 4930 | intel_dig_port->connected = bdw_digital_port_connected; | ||
| 4931 | else | ||
| 4932 | intel_dig_port->connected = lpt_digital_port_connected; | ||
| 4933 | } else { | ||
| 4934 | if (port == PORT_A) | ||
| 4935 | intel_dig_port->connected = hsw_digital_port_connected; | ||
| 4936 | else | ||
| 4937 | intel_dig_port->connected = lpt_digital_port_connected; | ||
| 4938 | } | ||
| 4939 | |||
| 4888 | intel_infoframe_init(intel_dig_port); | 4940 | intel_infoframe_init(intel_dig_port); |
| 4889 | 4941 | ||
| 4890 | return; | 4942 | return; |
| @@ -12866,6 +12866,16 @@ intel_dump_infoframe(struct drm_i915_private *dev_priv, · 12865 unmodified lines | |||
| 12866 | hdmi_infoframe_log(KERN_DEBUG, dev_priv->drm.dev, frame); | 12866 | hdmi_infoframe_log(KERN_DEBUG, dev_priv->drm.dev, frame); |
| 12867 | } | 12867 | } |
| 12868 | 12868 | ||
| 12869 | static void | ||
| 12870 | intel_dump_dp_vsc_sdp(struct drm_i915_private *dev_priv, | ||
| 12871 | const struct drm_dp_vsc_sdp *vsc) | ||
| 12872 | { | ||
| 12873 | if (!drm_debug_enabled(DRM_UT_KMS)) | ||
| 12874 | return; | ||
| 12875 | |||
| 12876 | drm_dp_vsc_sdp_log(KERN_DEBUG, dev_priv->drm.dev, vsc); | ||
| 12877 | } | ||
| 12878 | |||
| 12869 | #define OUTPUT_TYPE(x) [INTEL_OUTPUT_ ## x] = #x | 12879 | #define OUTPUT_TYPE(x) [INTEL_OUTPUT_ ## x] = #x |
| 12870 | 12880 | ||
| 12871 | static const char * const output_type_str[] = { | 12881 | static const char * const output_type_str[] = { |
| @@ -13023,6 +13033,15 @@ static void intel_dump_pipe_config(const struct intel_crtc_state *pipe_config, · 151 unmodified lines | |||
| 13023 | if (pipe_config->infoframes.enable & | 13033 | if (pipe_config->infoframes.enable & |
| 13024 | intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_VENDOR)) | 13034 | intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_VENDOR)) |
| 13025 | intel_dump_infoframe(dev_priv, &pipe_config->infoframes.hdmi); | 13035 | intel_dump_infoframe(dev_priv, &pipe_config->infoframes.hdmi); |
| 13036 | if (pipe_config->infoframes.enable & | ||
| 13037 | intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_DRM)) | ||
| 13038 | intel_dump_infoframe(dev_priv, &pipe_config->infoframes.drm); | ||
| 13039 | if (pipe_config->infoframes.enable & | ||
| 13040 | intel_hdmi_infoframe_enable(HDMI_PACKET_TYPE_GAMUT_METADATA)) | ||
| 13041 | intel_dump_infoframe(dev_priv, &pipe_config->infoframes.drm); | ||
| 13042 | if (pipe_config->infoframes.enable & | ||
| 13043 | intel_hdmi_infoframe_enable(DP_SDP_VSC)) | ||
| 13044 | intel_dump_dp_vsc_sdp(dev_priv, &pipe_config->infoframes.vsc); | ||
| 13026 | 13045 | ||
| 13027 | drm_dbg_kms(&dev_priv->drm, "requested mode:\n"); | 13046 | drm_dbg_kms(&dev_priv->drm, "requested mode:\n"); |
| 13028 | drm_mode_debug_printmodeline(&pipe_config->hw.mode); | 13047 | drm_mode_debug_printmodeline(&pipe_config->hw.mode); |
| @@ -13470,6 +13489,13 @@ intel_compare_infoframe(const union hdmi_infoframe *a, · 441 unmodified lines | |||
| 13470 | return memcmp(a, b, sizeof(*a)) == 0; | 13489 | return memcmp(a, b, sizeof(*a)) == 0; |
| 13471 | } | 13490 | } |
| 13472 | 13491 | ||
| 13492 | static bool | ||
| 13493 | intel_compare_dp_vsc_sdp(const struct drm_dp_vsc_sdp *a, | ||
| 13494 | const struct drm_dp_vsc_sdp *b) | ||
| 13495 | { | ||
| 13496 | return memcmp(a, b, sizeof(*a)) == 0; | ||
| 13497 | } | ||
| 13498 | |||
| 13473 | static void | 13499 | static void |
| 13474 | pipe_config_infoframe_mismatch(struct drm_i915_private *dev_priv, | 13500 | pipe_config_infoframe_mismatch(struct drm_i915_private *dev_priv, |
| 13475 | bool fastset, const char *name, | 13501 | bool fastset, const char *name, |
| @@ -13495,6 +13521,31 @@ pipe_config_infoframe_mismatch(struct drm_i915_private *dev_priv, · 19 unmodified lines | |||
| 13495 | } | 13521 | } |
| 13496 | } | 13522 | } |
| 13497 | 13523 | ||
| 13524 | static void | ||
| 13525 | pipe_config_dp_vsc_sdp_mismatch(struct drm_i915_private *dev_priv, | ||
| 13526 | bool fastset, const char *name, | ||
| 13527 | const struct drm_dp_vsc_sdp *a, | ||
| 13528 | const struct drm_dp_vsc_sdp *b) | ||
| 13529 | { | ||
| 13530 | if (fastset) { | ||
| 13531 | if (!drm_debug_enabled(DRM_UT_KMS)) | ||
| 13532 | return; | ||
| 13533 | |||
| 13534 | drm_dbg_kms(&dev_priv->drm, | ||
| 13535 | "fastset mismatch in %s dp sdp\n", name); | ||
| 13536 | drm_dbg_kms(&dev_priv->drm, "expected:\n"); | ||
| 13537 | drm_dp_vsc_sdp_log(KERN_DEBUG, dev_priv->drm.dev, a); | ||
| 13538 | drm_dbg_kms(&dev_priv->drm, "found:\n"); | ||
| 13539 | drm_dp_vsc_sdp_log(KERN_DEBUG, dev_priv->drm.dev, b); | ||
| 13540 | } else { | ||
| 13541 | drm_err(&dev_priv->drm, "mismatch in %s dp sdp\n", name); | ||
| 13542 | drm_err(&dev_priv->drm, "expected:\n"); | ||
| 13543 | drm_dp_vsc_sdp_log(KERN_ERR, dev_priv->drm.dev, a); | ||
| 13544 | drm_err(&dev_priv->drm, "found:\n"); | ||
| 13545 | drm_dp_vsc_sdp_log(KERN_ERR, dev_priv->drm.dev, b); | ||
| 13546 | } | ||
| 13547 | } | ||
| 13548 | |||
| 13498 | static void __printf(4, 5) | 13549 | static void __printf(4, 5) |
| 13499 | pipe_config_mismatch(bool fastset, const struct intel_crtc *crtc, | 13550 | pipe_config_mismatch(bool fastset, const struct intel_crtc *crtc, |
| 13500 | const char *name, const char *format, ...) | 13551 | const char *name, const char *format, ...) |
| @@ -13696,6 +13747,17 @@ intel_pipe_config_compare(const struct intel_crtc_state *current_config, · 195 unmodified lines | |||
| 13696 | } \ | 13747 | } \ |
| 13697 | } while (0) | 13748 | } while (0) |
| 13698 | 13749 | ||
| 13750 | #define PIPE_CONF_CHECK_DP_VSC_SDP(name) do { \ | ||
| 13751 | if (!current_config->has_psr && !pipe_config->has_psr && \ | ||
| 13752 | !intel_compare_dp_vsc_sdp(¤t_config->infoframes.name, \ | ||
| 13753 | &pipe_config->infoframes.name)) { \ | ||
| 13754 | pipe_config_dp_vsc_sdp_mismatch(dev_priv, fastset, __stringify(name), \ | ||
| 13755 | ¤t_config->infoframes.name, \ | ||
| 13756 | &pipe_config->infoframes.name); \ | ||
| 13757 | ret = false; \ | ||
| 13758 | } \ | ||
| 13759 | } while (0) | ||
| 13760 | |||
| 13699 | #define PIPE_CONF_CHECK_COLOR_LUT(name1, name2, bit_precision) do { \ | 13761 | #define PIPE_CONF_CHECK_COLOR_LUT(name1, name2, bit_precision) do { \ |
| 13700 | if (current_config->name1 != pipe_config->name1) { \ | 13762 | if (current_config->name1 != pipe_config->name1) { \ |
| 13701 | pipe_config_mismatch(fastset, crtc, __stringify(name1), \ | 13763 | pipe_config_mismatch(fastset, crtc, __stringify(name1), \ |
| @@ -13873,6 +13935,7 @@ intel_pipe_config_compare(const struct intel_crtc_state *current_config, · 171 unmodified lines | |||
| 13873 | PIPE_CONF_CHECK_INFOFRAME(spd); | 13935 | PIPE_CONF_CHECK_INFOFRAME(spd); |
| 13874 | PIPE_CONF_CHECK_INFOFRAME(hdmi); | 13936 | PIPE_CONF_CHECK_INFOFRAME(hdmi); |
| 13875 | PIPE_CONF_CHECK_INFOFRAME(drm); | 13937 | PIPE_CONF_CHECK_INFOFRAME(drm); |
| 13938 | PIPE_CONF_CHECK_DP_VSC_SDP(vsc); | ||
| 13876 | 13939 | ||
| 13877 | PIPE_CONF_CHECK_X(sync_mode_slaves_mask); | 13940 | PIPE_CONF_CHECK_X(sync_mode_slaves_mask); |
| 13878 | PIPE_CONF_CHECK_I(master_transcoder); | 13941 | PIPE_CONF_CHECK_I(master_transcoder); |
| @@ -13961,7 +14024,9 @@ static void verify_wm_state(struct intel_crtc *crtc, · 82 unmodified lines | |||
| 13961 | /* Watermarks */ | 14024 | /* Watermarks */ |
| 13962 | for (level = 0; level <= max_level; level++) { | 14025 | for (level = 0; level <= max_level; level++) { |
| 13963 | if (skl_wm_level_equals(&hw_plane_wm->wm[level], | 14026 | if (skl_wm_level_equals(&hw_plane_wm->wm[level], |
| 13964 | &sw_plane_wm->wm[level])) | 14027 | &sw_plane_wm->wm[level]) || |
| 14028 | (level == 0 && skl_wm_level_equals(&hw_plane_wm->wm[level], | ||
| 14029 | &sw_plane_wm->sagv_wm0))) | ||
| 13965 | continue; | 14030 | continue; |
| 13966 | 14031 | ||
| 13967 | drm_err(&dev_priv->drm, | 14032 | drm_err(&dev_priv->drm, |
| @@ -14016,7 +14081,9 @@ static void verify_wm_state(struct intel_crtc *crtc, · 48 unmodified lines | |||
| 14016 | /* Watermarks */ | 14081 | /* Watermarks */ |
| 14017 | for (level = 0; level <= max_level; level++) { | 14082 | for (level = 0; level <= max_level; level++) { |
| 14018 | if (skl_wm_level_equals(&hw_plane_wm->wm[level], | 14083 | if (skl_wm_level_equals(&hw_plane_wm->wm[level], |
| 14019 | &sw_plane_wm->wm[level])) | 14084 | &sw_plane_wm->wm[level]) || |
| 14085 | (level == 0 && skl_wm_level_equals(&hw_plane_wm->wm[level], | ||
| 14086 | &sw_plane_wm->sagv_wm0))) | ||
| 14020 | continue; | 14087 | continue; |
| 14021 | 14088 | ||
| 14022 | drm_err(&dev_priv->drm, | 14089 | drm_err(&dev_priv->drm, |
| @@ -15378,11 +15445,11 @@ static void intel_atomic_commit_tail(struct intel_atomic_state *state) · 1355 unmodified lines | |||
| 15378 | 15445 | ||
| 15379 | intel_set_cdclk_pre_plane_update(state); | 15446 | intel_set_cdclk_pre_plane_update(state); |
| 15380 | 15447 | ||
| 15381 | intel_sagv_pre_plane_update(state); | ||
| 15382 | |||
| 15383 | intel_modeset_verify_disabled(dev_priv, state); | 15448 | intel_modeset_verify_disabled(dev_priv, state); |
| 15384 | } | 15449 | } |
| 15385 | 15450 | ||
| 15451 | intel_sagv_pre_plane_update(state); | ||
| 15452 | |||
| 15386 | /* Complete the events for pipes that have now been disabled */ | 15453 | /* Complete the events for pipes that have now been disabled */ |
| 15387 | for_each_new_intel_crtc_in_state(state, crtc, new_crtc_state, i) { | 15454 | for_each_new_intel_crtc_in_state(state, crtc, new_crtc_state, i) { |
| 15388 | bool modeset = needs_modeset(new_crtc_state); | 15455 | bool modeset = needs_modeset(new_crtc_state); |
| @@ -15475,11 +15542,10 @@ static void intel_atomic_commit_tail(struct intel_atomic_state *state) · 86 unmodified lines | |||
| 15475 | intel_check_cpu_fifo_underruns(dev_priv); | 15542 | intel_check_cpu_fifo_underruns(dev_priv); |
| 15476 | intel_check_pch_fifo_underruns(dev_priv); | 15543 | intel_check_pch_fifo_underruns(dev_priv); |
| 15477 | 15544 | ||
| 15478 | if (state->modeset) { | 15545 | if (state->modeset) |
| 15479 | intel_verify_planes(state); | 15546 | intel_verify_planes(state); |
| 15480 | 15547 | ||
| 15481 | intel_sagv_post_plane_update(state); | 15548 | intel_sagv_post_plane_update(state); |
| 15482 | } | ||
| 15483 | 15549 | ||
| 15484 | drm_atomic_helper_commit_hw_done(&state->base); | 15550 | drm_atomic_helper_commit_hw_done(&state->base); |
| 15485 | 15551 | ||
| @@ -15815,7 +15881,7 @@ intel_prepare_plane_fb(struct drm_plane *_plane, · 329 unmodified lines | |||
| 15815 | if (new_plane_state->uapi.fence) { /* explicit fencing */ | 15881 | if (new_plane_state->uapi.fence) { /* explicit fencing */ |
| 15816 | ret = i915_sw_fence_await_dma_fence(&state->commit_ready, | 15882 | ret = i915_sw_fence_await_dma_fence(&state->commit_ready, |
| 15817 | new_plane_state->uapi.fence, | 15883 | new_plane_state->uapi.fence, |
| 15818 | I915_FENCE_TIMEOUT, | 15884 | i915_fence_timeout(dev_priv), |
| 15819 | GFP_KERNEL); | 15885 | GFP_KERNEL); |
| 15820 | if (ret < 0) | 15886 | if (ret < 0) |
| 15821 | return ret; | 15887 | return ret; |
| @@ -15842,7 +15908,8 @@ intel_prepare_plane_fb(struct drm_plane *_plane, · 20 unmodified lines | |||
| 15842 | 15908 | ||
| 15843 | ret = i915_sw_fence_await_reservation(&state->commit_ready, | 15909 | ret = i915_sw_fence_await_reservation(&state->commit_ready, |
| 15844 | obj->base.resv, NULL, | 15910 | obj->base.resv, NULL, |
| 15845 | false, I915_FENCE_TIMEOUT, | 15911 | false, |
| 15912 | i915_fence_timeout(dev_priv), | ||
| 15846 | GFP_KERNEL); | 15913 | GFP_KERNEL); |
| 15847 | if (ret < 0) | 15914 | if (ret < 0) |
| 15848 | goto unpin_fb; | 15915 | goto unpin_fb; |
| @@ -688,11 +688,13 @@ struct skl_plane_wm { · 687 unmodified lines | |||
| 688 | struct skl_wm_level wm[8]; | 688 | struct skl_wm_level wm[8]; |
| 689 | struct skl_wm_level uv_wm[8]; | 689 | struct skl_wm_level uv_wm[8]; |
| 690 | struct skl_wm_level trans_wm; | 690 | struct skl_wm_level trans_wm; |
| 691 | struct skl_wm_level sagv_wm0; | ||
| 691 | bool is_planar; | 692 | bool is_planar; |
| 692 | }; | 693 | }; |
| 693 | 694 | ||
| 694 | struct skl_pipe_wm { | 695 | struct skl_pipe_wm { |
| 695 | struct skl_plane_wm planes[I915_MAX_PLANES]; | 696 | struct skl_plane_wm planes[I915_MAX_PLANES]; |
| 697 | bool use_sagv_wm; | ||
| 696 | }; | 698 | }; |
| 697 | 699 | ||
| 698 | enum vlv_wm_level { | 700 | enum vlv_wm_level { |
| @@ -1424,6 +1426,7 @@ struct intel_digital_port { · 725 unmodified lines | |||
| 1424 | const struct drm_connector_state *conn_state); | 1426 | const struct drm_connector_state *conn_state); |
| 1425 | u32 (*infoframes_enabled)(struct intel_encoder *encoder, | 1427 | u32 (*infoframes_enabled)(struct intel_encoder *encoder, |
| 1426 | const struct intel_crtc_state *pipe_config); | 1428 | const struct intel_crtc_state *pipe_config); |
| 1429 | bool (*connected)(struct intel_encoder *encoder); | ||
| 1427 | }; | 1430 | }; |
| 1428 | 1431 | ||
| 1429 | struct intel_dp_mst_encoder { | 1432 | struct intel_dp_mst_encoder { |
| @@ -2487,8 +2487,8 @@ static void intel_dp_compute_vsc_sdp(struct intel_dp *intel_dp, · 2486 unmodified lines | |||
| 2487 | { | 2487 | { |
| 2488 | struct drm_dp_vsc_sdp *vsc = &crtc_state->infoframes.vsc; | 2488 | struct drm_dp_vsc_sdp *vsc = &crtc_state->infoframes.vsc; |
| 2489 | 2489 | ||
| 2490 | /* When PSR is enabled, VSC SDP is handled by PSR routine */ | 2490 | /* When a crtc state has PSR, VSC SDP will be handled by PSR routine */ |
| 2491 | if (intel_psr_enabled(intel_dp)) | 2491 | if (crtc_state->has_psr) |
| 2492 | return; | 2492 | return; |
| 2493 | 2493 | ||
| 2494 | if (!intel_dp_needs_vsc_sdp(crtc_state, conn_state)) | 2494 | if (!intel_dp_needs_vsc_sdp(crtc_state, conn_state)) |
| @@ -2500,6 +2500,42 @@ static void intel_dp_compute_vsc_sdp(struct intel_dp *intel_dp, · 5 unmodified lines | |||
| 2500 | &crtc_state->infoframes.vsc); | 2500 | &crtc_state->infoframes.vsc); |
| 2501 | } | 2501 | } |
| 2502 | 2502 | ||
| 2503 | void intel_dp_compute_psr_vsc_sdp(struct intel_dp *intel_dp, | ||
| 2504 | const struct intel_crtc_state *crtc_state, | ||
| 2505 | const struct drm_connector_state *conn_state, | ||
| 2506 | struct drm_dp_vsc_sdp *vsc) | ||
| 2507 | { | ||
| 2508 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); | ||
| 2509 | |||
| 2510 | vsc->sdp_type = DP_SDP_VSC; | ||
| 2511 | |||
| 2512 | if (dev_priv->psr.psr2_enabled) { | ||
| 2513 | if (dev_priv->psr.colorimetry_support && | ||
| 2514 | intel_dp_needs_vsc_sdp(crtc_state, conn_state)) { | ||
| 2515 | /* [PSR2, +Colorimetry] */ | ||
| 2516 | intel_dp_compute_vsc_colorimetry(crtc_state, conn_state, | ||
| 2517 | vsc); | ||
| 2518 | } else { | ||
| 2519 | /* | ||
| 2520 | * [PSR2, -Colorimetry] | ||
| 2521 | * Prepare VSC Header for SU as per eDP 1.4 spec, Table 6-11 | ||
| 2522 | * 3D stereo + PSR/PSR2 + Y-coordinate. | ||
| 2523 | */ | ||
| 2524 | vsc->revision = 0x4; | ||
| 2525 | vsc->length = 0xe; | ||
| 2526 | } | ||
| 2527 | } else { | ||
| 2528 | /* | ||
| 2529 | * [PSR1] | ||
| 2530 | * Prepare VSC Header for SU as per DP 1.4 spec, Table 2-118 | ||
| 2531 | * VSC SDP supporting 3D stereo + PSR (applies to eDP v1.3 or | ||
| 2532 | * higher). | ||
| 2533 | */ | ||
| 2534 | vsc->revision = 0x2; | ||
| 2535 | vsc->length = 0x8; | ||
| 2536 | } | ||
| 2537 | } | ||
| 2538 | |||
| 2503 | static void | 2539 | static void |
| 2504 | intel_dp_compute_hdr_metadata_infoframe_sdp(struct intel_dp *intel_dp, | 2540 | intel_dp_compute_hdr_metadata_infoframe_sdp(struct intel_dp *intel_dp, |
| 2505 | struct intel_crtc_state *crtc_state, | 2541 | struct intel_crtc_state *crtc_state, |
| @@ -4791,6 +4827,13 @@ static ssize_t intel_dp_vsc_sdp_pack(const struct drm_dp_vsc_sdp *vsc, · 2285 unmodified lines | |||
| 4791 | sdp->sdp_header.HB2 = vsc->revision; /* Revision Number */ | 4827 | sdp->sdp_header.HB2 = vsc->revision; /* Revision Number */ |
| 4792 | sdp->sdp_header.HB3 = vsc->length; /* Number of Valid Data Bytes */ | 4828 | sdp->sdp_header.HB3 = vsc->length; /* Number of Valid Data Bytes */ |
| 4793 | 4829 | ||
| 4830 | /* | ||
| 4831 | * Only revision 0x5 supports Pixel Encoding/Colorimetry Format as | ||
| 4832 | * per DP 1.4a spec. | ||
| 4833 | */ | ||
| 4834 | if (vsc->revision != 0x5) | ||
| 4835 | goto out; | ||
| 4836 | |||
| 4794 | /* VSC SDP Payload for DB16 through DB18 */ | 4837 | /* VSC SDP Payload for DB16 through DB18 */ |
| 4795 | /* Pixel Encoding and Colorimetry Formats */ | 4838 | /* Pixel Encoding and Colorimetry Formats */ |
| 4796 | sdp->db[16] = (vsc->pixelformat & 0xf) << 4; /* DB16[7:4] */ | 4839 | sdp->db[16] = (vsc->pixelformat & 0xf) << 4; /* DB16[7:4] */ |
| @@ -4823,6 +4866,7 @@ static ssize_t intel_dp_vsc_sdp_pack(const struct drm_dp_vsc_sdp *vsc, · 26 unmodified lines | |||
| 4823 | /* Content Type */ | 4866 | /* Content Type */ |
| 4824 | sdp->db[18] = vsc->content_type & 0x7; | 4867 | sdp->db[18] = vsc->content_type & 0x7; |
| 4825 | 4868 | ||
| 4869 | out: | ||
| 4826 | return length; | 4870 | return length; |
| 4827 | } | 4871 | } |
| 4828 | 4872 | ||
| @@ -4935,6 +4979,24 @@ static void intel_write_dp_sdp(struct intel_encoder *encoder, · 106 unmodified lines | |||
| 4935 | intel_dig_port->write_infoframe(encoder, crtc_state, type, &sdp, len); | 4979 | intel_dig_port->write_infoframe(encoder, crtc_state, type, &sdp, len); |
| 4936 | } | 4980 | } |
| 4937 | 4981 | ||
| 4982 | void intel_write_dp_vsc_sdp(struct intel_encoder *encoder, | ||
| 4983 | const struct intel_crtc_state *crtc_state, | ||
| 4984 | struct drm_dp_vsc_sdp *vsc) | ||
| 4985 | { | ||
| 4986 | struct intel_digital_port *intel_dig_port = enc_to_dig_port(encoder); | ||
| 4987 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 4988 | struct dp_sdp sdp = {}; | ||
| 4989 | ssize_t len; | ||
| 4990 | |||
| 4991 | len = intel_dp_vsc_sdp_pack(vsc, &sdp, sizeof(sdp)); | ||
| 4992 | |||
| 4993 | if (drm_WARN_ON(&dev_priv->drm, len < 0)) | ||
| 4994 | return; | ||
| 4995 | |||
| 4996 | intel_dig_port->write_infoframe(encoder, crtc_state, DP_SDP_VSC, | ||
| 4997 | &sdp, len); | ||
| 4998 | } | ||
| 4999 | |||
| 4938 | void intel_dp_set_infoframes(struct intel_encoder *encoder, | 5000 | void intel_dp_set_infoframes(struct intel_encoder *encoder, |
| 4939 | bool enable, | 5001 | bool enable, |
| 4940 | const struct intel_crtc_state *crtc_state, | 5002 | const struct intel_crtc_state *crtc_state, |
| @@ -4971,233 +5033,191 @@ void intel_dp_set_infoframes(struct intel_encoder *encoder, · 30 unmodified lines | |||
| 4971 | intel_write_dp_sdp(encoder, crtc_state, HDMI_PACKET_TYPE_GAMUT_METADATA); | 5033 | intel_write_dp_sdp(encoder, crtc_state, HDMI_PACKET_TYPE_GAMUT_METADATA); |
| 4972 | } | 5034 | } |
| 4973 | 5035 | ||
| 4974 | static void | 5036 | static int intel_dp_vsc_sdp_unpack(struct drm_dp_vsc_sdp *vsc, |
| 4975 | intel_dp_setup_vsc_sdp(struct intel_dp *intel_dp, | 5037 | const void *buffer, size_t size) |
| 4976 | const struct intel_crtc_state *crtc_state, | ||
| 4977 | const struct drm_connector_state *conn_state) | ||
| 4978 | { | 5038 | { |
| 4979 | struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp); | 5039 | const struct dp_sdp *sdp = buffer; |
| 4980 | struct dp_sdp vsc_sdp = {}; | ||
| 4981 | |||
| 4982 | /* Prepare VSC Header for SU as per DP 1.4a spec, Table 2-119 */ | ||
| 4983 | vsc_sdp.sdp_header.HB0 = 0; | ||
| 4984 | vsc_sdp.sdp_header.HB1 = 0x7; | ||
| 4985 | 5040 | ||
| 4986 | /* | 5041 | if (size < sizeof(struct dp_sdp)) |
| 4987 | * VSC SDP supporting 3D stereo, PSR2, and Pixel Encoding/ | 5042 | return -EINVAL; |
| 4988 | * Colorimetry Format indication. | ||
| 4989 | */ | ||
| 4990 | vsc_sdp.sdp_header.HB2 = 0x5; | ||
| 4991 | 5043 | ||
| 4992 | /* | 5044 | memset(vsc, 0, size); |
| 4993 | * VSC SDP supporting 3D stereo, + PSR2, + Pixel Encoding/ | ||
| 4994 | * Colorimetry Format indication (HB2 = 05h). | ||
| 4995 | */ | ||
| 4996 | vsc_sdp.sdp_header.HB3 = 0x13; | ||
| 4997 | 5045 | ||
| 4998 | /* DP 1.4a spec, Table 2-120 */ | 5046 | if (sdp->sdp_header.HB0 != 0) |
| 4999 | switch (crtc_state->output_format) { | 5047 | return -EINVAL; |
| 5000 | case INTEL_OUTPUT_FORMAT_YCBCR444: | ||
| 5001 | vsc_sdp.db[16] = 0x1 << 4; /* YCbCr 444 : DB16[7:4] = 1h */ | ||
| 5002 | break; | ||
| 5003 | case INTEL_OUTPUT_FORMAT_YCBCR420: | ||
| 5004 | vsc_sdp.db[16] = 0x3 << 4; /* YCbCr 420 : DB16[7:4] = 3h */ | ||
| 5005 | break; | ||
| 5006 | case INTEL_OUTPUT_FORMAT_RGB: | ||
| 5007 | default: | ||
| 5008 | /* RGB: DB16[7:4] = 0h */ | ||
| 5009 | break; | ||
| 5010 | } | ||
| 5011 | 5048 | ||
| 5012 | switch (conn_state->colorspace) { | 5049 | if (sdp->sdp_header.HB1 != DP_SDP_VSC) |
| 5013 | case DRM_MODE_COLORIMETRY_BT709_YCC: | 5050 | return -EINVAL; |
| 5014 | vsc_sdp.db[16] |= 0x1; | ||
| 5015 | break; | ||
| 5016 | case DRM_MODE_COLORIMETRY_XVYCC_601: | ||
| 5017 | vsc_sdp.db[16] |= 0x2; | ||
| 5018 | break; | ||
| 5019 | case DRM_MODE_COLORIMETRY_XVYCC_709: | ||
| 5020 | vsc_sdp.db[16] |= 0x3; | ||
| 5021 | break; | ||
| 5022 | case DRM_MODE_COLORIMETRY_SYCC_601: | ||
| 5023 | vsc_sdp.db[16] |= 0x4; | ||
| 5024 | break; | ||
| 5025 | case DRM_MODE_COLORIMETRY_OPYCC_601: | ||
| 5026 | vsc_sdp.db[16] |= 0x5; | ||
| 5027 | break; | ||
| 5028 | case DRM_MODE_COLORIMETRY_BT2020_CYCC: | ||
| 5029 | case DRM_MODE_COLORIMETRY_BT2020_RGB: | ||
| 5030 | vsc_sdp.db[16] |= 0x6; | ||
| 5031 | break; | ||
| 5032 | case DRM_MODE_COLORIMETRY_BT2020_YCC: | ||
| 5033 | vsc_sdp.db[16] |= 0x7; | ||
| 5034 | break; | ||
| 5035 | case DRM_MODE_COLORIMETRY_DCI_P3_RGB_D65: | ||
| 5036 | case DRM_MODE_COLORIMETRY_DCI_P3_RGB_THEATER: | ||
| 5037 | vsc_sdp.db[16] |= 0x4; /* DCI-P3 (SMPTE RP 431-2) */ | ||
| 5038 | break; | ||
| 5039 | default: | ||
| 5040 | /* sRGB (IEC 61966-2-1) / ITU-R BT.601: DB16[0:3] = 0h */ | ||
| 5041 | 5051 | ||
| 5042 | /* RGB->YCBCR color conversion uses the BT.709 color space. */ | 5052 | vsc->sdp_type = sdp->sdp_header.HB1; |
| 5043 | if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR420) | 5053 | vsc->revision = sdp->sdp_header.HB2; |
| 5044 | vsc_sdp.db[16] |= 0x1; /* 0x1, ITU-R BT.709 */ | 5054 | vsc->length = sdp->sdp_header.HB3; |
| 5045 | break; | ||
| 5046 | } | ||
| 5047 | 5055 | ||
| 5048 | /* | 5056 | if ((sdp->sdp_header.HB2 == 0x2 && sdp->sdp_header.HB3 == 0x8) || |
| 5049 | * For pixel encoding formats YCbCr444, YCbCr422, YCbCr420, and Y Only, | 5057 | (sdp->sdp_header.HB2 == 0x4 && sdp->sdp_header.HB3 == 0xe)) { |
| 5050 | * the following Component Bit Depth values are defined: | 5058 | /* |
| 5051 | * 001b = 8bpc. | 5059 | * - HB2 = 0x2, HB3 = 0x8 |
| 5052 | * 010b = 10bpc. | 5060 | * VSC SDP supporting 3D stereo + PSR |
| 5053 | * 011b = 12bpc. | 5061 | * - HB2 = 0x4, HB3 = 0xe |
| 5054 | * 100b = 16bpc. | 5062 | * VSC SDP supporting 3D stereo + PSR2 with Y-coordinate of |
| 5055 | */ | 5063 | * first scan line of the SU region (applies to eDP v1.4b |
| 5056 | switch (crtc_state->pipe_bpp) { | 5064 | * and higher). |
| 5057 | case 24: /* 8bpc */ | 5065 | */ |
| 5058 | vsc_sdp.db[17] = 0x1; | 5066 | return 0; |
| 5059 | break; | 5067 | } else if (sdp->sdp_header.HB2 == 0x5 && sdp->sdp_header.HB3 == 0x13) { |
| 5060 | case 30: /* 10bpc */ | 5068 | /* |
| 5061 | vsc_sdp.db[17] = 0x2; | 5069 | * - HB2 = 0x5, HB3 = 0x13 |
| 5062 | break; | 5070 | * VSC SDP supporting 3D stereo + PSR2 + Pixel Encoding/Colorimetry |
| 5063 | case 36: /* 12bpc */ | 5071 | * Format. |
| 5064 | vsc_sdp.db[17] = 0x3; | 5072 | */ |
| 5065 | break; | 5073 | vsc->pixelformat = (sdp->db[16] >> 4) & 0xf; |
| 5066 | case 48: /* 16bpc */ | 5074 | vsc->colorimetry = sdp->db[16] & 0xf; |
| 5067 | vsc_sdp.db[17] = 0x4; | 5075 | vsc->dynamic_range = (sdp->db[17] >> 7) & 0x1; |
| 5068 | break; | ||
| 5069 | default: | ||
| 5070 | MISSING_CASE(crtc_state->pipe_bpp); | ||
| 5071 | break; | ||
| 5072 | } | ||
| 5073 | 5076 | ||
| 5074 | /* | 5077 | switch (sdp->db[17] & 0x7) { |
| 5075 | * Dynamic Range (Bit 7) | 5078 | case 0x0: |
| 5076 | * 0 = VESA range, 1 = CTA range. | 5079 | vsc->bpc = 6; |
| 5077 | * all YCbCr are always limited range | 5080 | break; |
| 5078 | */ | 5081 | case 0x1: |
| 5079 | vsc_sdp.db[17] |= 0x80; | 5082 | vsc->bpc = 8; |
| 5083 | break; | ||
| 5084 | case 0x2: | ||
| 5085 | vsc->bpc = 10; | ||
| 5086 | break; | ||
| 5087 | case 0x3: | ||
| 5088 | vsc->bpc = 12; | ||
| 5089 | break; | ||
| 5090 | case 0x4: | ||
| 5091 | vsc->bpc = 16; | ||
| 5092 | break; | ||
| 5093 | default: | ||
| 5094 | MISSING_CASE(sdp->db[17] & 0x7); | ||
| 5095 | return -EINVAL; | ||
| 5096 | } | ||
| 5080 | 5097 | ||
| 5081 | /* | 5098 | vsc->content_type = sdp->db[18] & 0x7; |
| 5082 | * Content Type (Bits 2:0) | 5099 | } else { |
| 5083 | * 000b = Not defined. | 5100 | return -EINVAL; |
| 5084 | * 001b = Graphics. | 5101 | } |
| 5085 | * 010b = Photo. | ||
| 5086 | * 011b = Video. | ||
| 5087 | * 100b = Game | ||
| 5088 | * All other values are RESERVED. | ||
| 5089 | * Note: See CTA-861-G for the definition and expected | ||
| 5090 | * processing by a stream sink for the above contect types. | ||
| 5091 | */ | ||
| 5092 | vsc_sdp.db[18] = 0; | ||
| 5093 | 5102 | ||
| 5094 | intel_dig_port->write_infoframe(&intel_dig_port->base, | 5103 | return 0; |
| 5095 | crtc_state, DP_SDP_VSC, &vsc_sdp, sizeof(vsc_sdp)); | ||
| 5096 | } | 5104 | } |
| 5097 | 5105 | ||
| 5098 | static void | 5106 | static int |
| 5099 | intel_dp_setup_hdr_metadata_infoframe_sdp(struct intel_dp *intel_dp, | 5107 | intel_dp_hdr_metadata_infoframe_sdp_unpack(struct hdmi_drm_infoframe *drm_infoframe, |
| 5100 | const struct intel_crtc_state *crtc_state, | 5108 | const void *buffer, size_t size) |
| 5101 | const struct drm_connector_state *conn_state) | ||
| 5102 | { | 5109 | { |
| 5103 | struct drm_i915_private *i915 = dp_to_i915(intel_dp); | ||
| 5104 | struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp); | ||
| 5105 | struct dp_sdp infoframe_sdp = {}; | ||
| 5106 | struct hdmi_drm_infoframe drm_infoframe = {}; | ||
| 5107 | const int infoframe_size = HDMI_INFOFRAME_HEADER_SIZE + HDMI_DRM_INFOFRAME_SIZE; | ||
| 5108 | unsigned char buf[HDMI_INFOFRAME_HEADER_SIZE + HDMI_DRM_INFOFRAME_SIZE]; | ||
| 5109 | ssize_t len; | ||
| 5110 | int ret; | 5110 | int ret; |
| 5111 | 5111 | ||
| 5112 | ret = drm_hdmi_infoframe_set_hdr_metadata(&drm_infoframe, conn_state); | 5112 | const struct dp_sdp *sdp = buffer; |
| 5113 | if (ret) { | ||
| 5114 | drm_dbg_kms(&i915->drm, | ||
| 5115 | "couldn't set HDR metadata in infoframe\n"); | ||
| 5116 | return; | ||
| 5117 | } | ||
| 5118 | 5113 | ||
| 5119 | len = hdmi_drm_infoframe_pack_only(&drm_infoframe, buf, sizeof(buf)); | 5114 | if (size < sizeof(struct dp_sdp)) |
| 5120 | if (len < 0) { | 5115 | return -EINVAL; |
| 5121 | drm_dbg_kms(&i915->drm, | ||
| 5122 | "buffer size is smaller than hdr metadata infoframe\n"); | ||
| 5123 | return; | ||
| 5124 | } | ||
| 5125 | 5116 | ||
| 5126 | if (len != infoframe_size) { | 5117 | if (sdp->sdp_header.HB0 != 0) |
| 5127 | drm_dbg_kms(&i915->drm, "wrong static hdr metadata size\n"); | 5118 | return -EINVAL; |
| 5128 | return; | ||
| 5129 | } | ||
| 5130 | 5119 | ||
| 5131 | /* | 5120 | if (sdp->sdp_header.HB1 != HDMI_INFOFRAME_TYPE_DRM) |
| 5132 | * Set up the infoframe sdp packet for HDR static metadata. | 5121 | return -EINVAL; |
| 5133 | * Prepare VSC Header for SU as per DP 1.4a spec, | ||
| 5134 | * Table 2-100 and Table 2-101 | ||
| 5135 | */ | ||
| 5136 | 5122 | ||
| 5137 | /* Packet ID, 00h for non-Audio INFOFRAME */ | ||
| 5138 | infoframe_sdp.sdp_header.HB0 = 0; | ||
| 5139 | /* | ||
| 5140 | * Packet Type 80h + Non-audio INFOFRAME Type value | ||
| 5141 | * HDMI_INFOFRAME_TYPE_DRM: 0x87, | ||
| 5142 | */ | ||
| 5143 | infoframe_sdp.sdp_header.HB1 = drm_infoframe.type; | ||
| 5144 | /* | 5123 | /* |
| 5145 | * Least Significant Eight Bits of (Data Byte Count – 1) | 5124 | * Least Significant Eight Bits of (Data Byte Count – 1) |
| 5146 | * infoframe_size - 1, | 5125 | * 1Dh (i.e., Data Byte Count = 30 bytes). |
| 5147 | */ | 5126 | */ |
| 5148 | infoframe_sdp.sdp_header.HB2 = 0x1D; | 5127 | if (sdp->sdp_header.HB2 != 0x1D) |
| 5128 | return -EINVAL; | ||
| 5129 | |||
| 5130 | /* Most Significant Two Bits of (Data Byte Count – 1), Clear to 00b. */ | ||
| 5131 | if ((sdp->sdp_header.HB3 & 0x3) != 0) | ||
| 5132 | return -EINVAL; | ||
| 5133 | |||
| 5149 | /* INFOFRAME SDP Version Number */ | 5134 | /* INFOFRAME SDP Version Number */ |
| 5150 | infoframe_sdp.sdp_header.HB3 = (0x13 << 2); | 5135 | if (((sdp->sdp_header.HB3 >> 2) & 0x3f) != 0x13) |
| 5136 | return -EINVAL; | ||
| 5137 | |||
| 5151 | /* CTA Header Byte 2 (INFOFRAME Version Number) */ | 5138 | /* CTA Header Byte 2 (INFOFRAME Version Number) */ |
| 5152 | infoframe_sdp.db[0] = drm_infoframe.version; | 5139 | if (sdp->db[0] != 1) |
| 5140 | return -EINVAL; | ||
| 5141 | |||
| 5153 | /* CTA Header Byte 3 (Length of INFOFRAME): HDMI_DRM_INFOFRAME_SIZE */ | 5142 | /* CTA Header Byte 3 (Length of INFOFRAME): HDMI_DRM_INFOFRAME_SIZE */ |
| 5154 | infoframe_sdp.db[1] = drm_infoframe.length; | 5143 | if (sdp->db[1] != HDMI_DRM_INFOFRAME_SIZE) |
| 5155 | /* | 5144 | return -EINVAL; |
| 5156 | * Copy HDMI_DRM_INFOFRAME_SIZE size from a buffer after | ||
| 5157 | * HDMI_INFOFRAME_HEADER_SIZE | ||
| 5158 | */ | ||
| 5159 | BUILD_BUG_ON(sizeof(infoframe_sdp.db) < HDMI_DRM_INFOFRAME_SIZE + 2); | ||
| 5160 | memcpy(&infoframe_sdp.db[2], &buf[HDMI_INFOFRAME_HEADER_SIZE], | ||
| 5161 | HDMI_DRM_INFOFRAME_SIZE); | ||
| 5162 | 5145 | ||
| 5163 | /* | 5146 | ret = hdmi_drm_infoframe_unpack_only(drm_infoframe, &sdp->db[2], |
| 5164 | * Size of DP infoframe sdp packet for HDR static metadata is consist of | 5147 | HDMI_DRM_INFOFRAME_SIZE); |
| 5165 | * - DP SDP Header(struct dp_sdp_header): 4 bytes | 5148 | |
| 5166 | * - Two Data Blocks: 2 bytes | 5149 | return ret; |
| 5167 | * CTA Header Byte2 (INFOFRAME Version Number) | ||
| 5168 | * CTA Header Byte3 (Length of INFOFRAME) | ||
| 5169 | * - HDMI_DRM_INFOFRAME_SIZE: 26 bytes | ||
| 5170 | * | ||
| 5171 | * Prior to GEN11's GMP register size is identical to DP HDR static metadata | ||
| 5172 | * infoframe size. But GEN11+ has larger than that size, write_infoframe | ||
| 5173 | * will pad rest of the size. | ||
| 5174 | */ | ||
| 5175 | intel_dig_port->write_infoframe(&intel_dig_port->base, crtc_state, | ||
| 5176 | HDMI_PACKET_TYPE_GAMUT_METADATA, | ||
| 5177 | &infoframe_sdp, | ||
| 5178 | sizeof(struct dp_sdp_header) + 2 + HDMI_DRM_INFOFRAME_SIZE); | ||
| 5179 | } | 5150 | } |
| 5180 | 5151 | ||
| 5181 | void intel_dp_vsc_enable(struct intel_dp *intel_dp, | 5152 | static void intel_read_dp_vsc_sdp(struct intel_encoder *encoder, |
| 5182 | const struct intel_crtc_state *crtc_state, | 5153 | struct intel_crtc_state *crtc_state, |
| 5183 | const struct drm_connector_state *conn_state) | 5154 | struct drm_dp_vsc_sdp *vsc) |
| 5184 | { | 5155 | { |
| 5185 | if (!intel_dp_needs_vsc_sdp(crtc_state, conn_state)) | 5156 | struct intel_digital_port *intel_dig_port = enc_to_dig_port(encoder); |
| 5157 | struct intel_dp *intel_dp = enc_to_intel_dp(encoder); | ||
| 5158 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 5159 | unsigned int type = DP_SDP_VSC; | ||
| 5160 | struct dp_sdp sdp = {}; | ||
| 5161 | int ret; | ||
| 5162 | |||
| 5163 | /* When PSR is enabled, VSC SDP is handled by PSR routine */ | ||
| 5164 | if (intel_psr_enabled(intel_dp)) | ||
| 5165 | return; | ||
| 5166 | |||
| 5167 | if ((crtc_state->infoframes.enable & | ||
| 5168 | intel_hdmi_infoframe_enable(type)) == 0) | ||
| 5186 | return; | 5169 | return; |
| 5187 | 5170 | ||
| 5188 | intel_dp_setup_vsc_sdp(intel_dp, crtc_state, conn_state); | 5171 | intel_dig_port->read_infoframe(encoder, crtc_state, type, &sdp, sizeof(sdp)); |
| 5172 | |||
| 5173 | ret = intel_dp_vsc_sdp_unpack(vsc, &sdp, sizeof(sdp)); | ||
| 5174 | |||
| 5175 | if (ret) | ||
| 5176 | drm_dbg_kms(&dev_priv->drm, "Failed to unpack DP VSC SDP\n"); | ||
| 5189 | } | 5177 | } |
| 5190 | 5178 | ||
| 5191 | void intel_dp_hdr_metadata_enable(struct intel_dp *intel_dp, | 5179 | static void intel_read_dp_hdr_metadata_infoframe_sdp(struct intel_encoder *encoder, |
| 5192 | const struct intel_crtc_state *crtc_state, | 5180 | struct intel_crtc_state *crtc_state, |
| 5193 | const struct drm_connector_state *conn_state) | 5181 | struct hdmi_drm_infoframe *drm_infoframe) |
| 5194 | { | 5182 | { |
| 5195 | if (!conn_state->hdr_output_metadata) | 5183 | struct intel_digital_port *intel_dig_port = enc_to_dig_port(encoder); |
| 5184 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 5185 | unsigned int type = HDMI_PACKET_TYPE_GAMUT_METADATA; | ||
| 5186 | struct dp_sdp sdp = {}; | ||
| 5187 | int ret; | ||
| 5188 | |||
| 5189 | if ((crtc_state->infoframes.enable & | ||
| 5190 | intel_hdmi_infoframe_enable(type)) == 0) | ||
| 5196 | return; | 5191 | return; |
| 5197 | 5192 | ||
| 5198 | intel_dp_setup_hdr_metadata_infoframe_sdp(intel_dp, | 5193 | intel_dig_port->read_infoframe(encoder, crtc_state, type, &sdp, |
| 5199 | crtc_state, | 5194 | sizeof(sdp)); |
| 5200 | conn_state); | 5195 | |
| 5196 | ret = intel_dp_hdr_metadata_infoframe_sdp_unpack(drm_infoframe, &sdp, | ||
| 5197 | sizeof(sdp)); | ||
| 5198 | |||
| 5199 | if (ret) | ||
| 5200 | drm_dbg_kms(&dev_priv->drm, | ||
| 5201 | "Failed to unpack DP HDR Metadata Infoframe SDP\n"); | ||
| 5202 | } | ||
| 5203 | |||
| 5204 | void intel_read_dp_sdp(struct intel_encoder *encoder, | ||
| 5205 | struct intel_crtc_state *crtc_state, | ||
| 5206 | unsigned int type) | ||
| 5207 | { | ||
| 5208 | switch (type) { | ||
| 5209 | case DP_SDP_VSC: | ||
| 5210 | intel_read_dp_vsc_sdp(encoder, crtc_state, | ||
| 5211 | &crtc_state->infoframes.vsc); | ||
| 5212 | break; | ||
| 5213 | case HDMI_PACKET_TYPE_GAMUT_METADATA: | ||
| 5214 | intel_read_dp_hdr_metadata_infoframe_sdp(encoder, crtc_state, | ||
| 5215 | &crtc_state->infoframes.drm.drm); | ||
| 5216 | break; | ||
| 5217 | default: | ||
| 5218 | MISSING_CASE(type); | ||
| 5219 | break; | ||
| 5220 | } | ||
| 5201 | } | 5221 | } |
| 5202 | 5222 | ||
| 5203 | static u8 intel_dp_autotest_link_training(struct intel_dp *intel_dp) | 5223 | static u8 intel_dp_autotest_link_training(struct intel_dp *intel_dp) |
| @@ -5998,64 +6018,7 @@ edp_detect(struct intel_dp *intel_dp) · 794 unmodified lines | |||
| 5998 | static bool ibx_digital_port_connected(struct intel_encoder *encoder) | 6018 | static bool ibx_digital_port_connected(struct intel_encoder *encoder) |
| 5999 | { | 6019 | { |
| 6000 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | 6020 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); |
| 6001 | u32 bit; | 6021 | u32 bit = dev_priv->hotplug.pch_hpd[encoder->hpd_pin]; |
| 6002 | |||
| 6003 | switch (encoder->hpd_pin) { | ||
| 6004 | case HPD_PORT_B: | ||
| 6005 | bit = SDE_PORTB_HOTPLUG; | ||
| 6006 | break; | ||
| 6007 | case HPD_PORT_C: | ||
| 6008 | bit = SDE_PORTC_HOTPLUG; | ||
| 6009 | break; | ||
| 6010 | case HPD_PORT_D: | ||
| 6011 | bit = SDE_PORTD_HOTPLUG; | ||
| 6012 | break; | ||
| 6013 | default: | ||
| 6014 | MISSING_CASE(encoder->hpd_pin); | ||
| 6015 | return false; | ||
| 6016 | } | ||
| 6017 | |||
| 6018 | return intel_de_read(dev_priv, SDEISR) & bit; | ||
| 6019 | } | ||
| 6020 | |||
| 6021 | static bool cpt_digital_port_connected(struct intel_encoder *encoder) | ||
| 6022 | { | ||
| 6023 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6024 | u32 bit; | ||
| 6025 | |||
| 6026 | switch (encoder->hpd_pin) { | ||
| 6027 | case HPD_PORT_B: | ||
| 6028 | bit = SDE_PORTB_HOTPLUG_CPT; | ||
| 6029 | break; | ||
| 6030 | case HPD_PORT_C: | ||
| 6031 | bit = SDE_PORTC_HOTPLUG_CPT; | ||
| 6032 | break; | ||
| 6033 | case HPD_PORT_D: | ||
| 6034 | bit = SDE_PORTD_HOTPLUG_CPT; | ||
| 6035 | break; | ||
| 6036 | default: | ||
| 6037 | MISSING_CASE(encoder->hpd_pin); | ||
| 6038 | return false; | ||
| 6039 | } | ||
| 6040 | |||
| 6041 | return intel_de_read(dev_priv, SDEISR) & bit; | ||
| 6042 | } | ||
| 6043 | |||
| 6044 | static bool spt_digital_port_connected(struct intel_encoder *encoder) | ||
| 6045 | { | ||
| 6046 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6047 | u32 bit; | ||
| 6048 | |||
| 6049 | switch (encoder->hpd_pin) { | ||
| 6050 | case HPD_PORT_A: | ||
| 6051 | bit = SDE_PORTA_HOTPLUG_SPT; | ||
| 6052 | break; | ||
| 6053 | case HPD_PORT_E: | ||
| 6054 | bit = SDE_PORTE_HOTPLUG_SPT; | ||
| 6055 | break; | ||
| 6056 | default: | ||
| 6057 | return cpt_digital_port_connected(encoder); | ||
| 6058 | } | ||
| 6059 | 6022 | ||
| 6060 | return intel_de_read(dev_priv, SDEISR) & bit; | 6023 | return intel_de_read(dev_priv, SDEISR) & bit; |
| 6061 | } | 6024 | } |
| @@ -6109,89 +6072,9 @@ static bool gm45_digital_port_connected(struct intel_encoder *encoder) · 47 unmodified lines | |||
| 6109 | static bool ilk_digital_port_connected(struct intel_encoder *encoder) | 6072 | static bool ilk_digital_port_connected(struct intel_encoder *encoder) |
| 6110 | { | 6073 | { |
| 6111 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | 6074 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); |
| 6075 | u32 bit = dev_priv->hotplug.hpd[encoder->hpd_pin]; | ||
| 6112 | 6076 | ||
| 6113 | if (encoder->hpd_pin == HPD_PORT_A) | 6077 | return intel_de_read(dev_priv, DEISR) & bit; |
| 6114 | return intel_de_read(dev_priv, DEISR) & DE_DP_A_HOTPLUG; | ||
| 6115 | else | ||
| 6116 | return ibx_digital_port_connected(encoder); | ||
| 6117 | } | ||
| 6118 | |||
| 6119 | static bool snb_digital_port_connected(struct intel_encoder *encoder) | ||
| 6120 | { | ||
| 6121 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6122 | |||
| 6123 | if (encoder->hpd_pin == HPD_PORT_A) | ||
| 6124 | return intel_de_read(dev_priv, DEISR) & DE_DP_A_HOTPLUG; | ||
| 6125 | else | ||
| 6126 | return cpt_digital_port_connected(encoder); | ||
| 6127 | } | ||
| 6128 | |||
| 6129 | static bool ivb_digital_port_connected(struct intel_encoder *encoder) | ||
| 6130 | { | ||
| 6131 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6132 | |||
| 6133 | if (encoder->hpd_pin == HPD_PORT_A) | ||
| 6134 | return intel_de_read(dev_priv, DEISR) & DE_DP_A_HOTPLUG_IVB; | ||
| 6135 | else | ||
| 6136 | return cpt_digital_port_connected(encoder); | ||
| 6137 | } | ||
| 6138 | |||
| 6139 | static bool bdw_digital_port_connected(struct intel_encoder *encoder) | ||
| 6140 | { | ||
| 6141 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6142 | |||
| 6143 | if (encoder->hpd_pin == HPD_PORT_A) | ||
| 6144 | return intel_de_read(dev_priv, GEN8_DE_PORT_ISR) & GEN8_PORT_DP_A_HOTPLUG; | ||
| 6145 | else | ||
| 6146 | return cpt_digital_port_connected(encoder); | ||
| 6147 | } | ||
| 6148 | |||
| 6149 | static bool bxt_digital_port_connected(struct intel_encoder *encoder) | ||
| 6150 | { | ||
| 6151 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6152 | u32 bit; | ||
| 6153 | |||
| 6154 | switch (encoder->hpd_pin) { | ||
| 6155 | case HPD_PORT_A: | ||
| 6156 | bit = BXT_DE_PORT_HP_DDIA; | ||
| 6157 | break; | ||
| 6158 | case HPD_PORT_B: | ||
| 6159 | bit = BXT_DE_PORT_HP_DDIB; | ||
| 6160 | break; | ||
| 6161 | case HPD_PORT_C: | ||
| 6162 | bit = BXT_DE_PORT_HP_DDIC; | ||
| 6163 | break; | ||
| 6164 | default: | ||
| 6165 | MISSING_CASE(encoder->hpd_pin); | ||
| 6166 | return false; | ||
| 6167 | } | ||
| 6168 | |||
| 6169 | return intel_de_read(dev_priv, GEN8_DE_PORT_ISR) & bit; | ||
| 6170 | } | ||
| 6171 | |||
| 6172 | static bool intel_combo_phy_connected(struct drm_i915_private *dev_priv, | ||
| 6173 | enum phy phy) | ||
| 6174 | { | ||
| 6175 | if (HAS_PCH_MCC(dev_priv) && phy == PHY_C) | ||
| 6176 | return intel_de_read(dev_priv, SDEISR) & SDE_TC_HOTPLUG_ICP(PORT_TC1); | ||
| 6177 | |||
| 6178 | return intel_de_read(dev_priv, SDEISR) & SDE_DDI_HOTPLUG_ICP(phy); | ||
| 6179 | } | ||
| 6180 | |||
| 6181 | static bool icp_digital_port_connected(struct intel_encoder *encoder) | ||
| 6182 | { | ||
| 6183 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6184 | struct intel_digital_port *dig_port = enc_to_dig_port(encoder); | ||
| 6185 | enum phy phy = intel_port_to_phy(dev_priv, encoder->port); | ||
| 6186 | |||
| 6187 | if (intel_phy_is_combo(dev_priv, phy)) | ||
| 6188 | return intel_combo_phy_connected(dev_priv, phy); | ||
| 6189 | else if (intel_phy_is_tc(dev_priv, phy)) | ||
| 6190 | return intel_tc_port_connected(dig_port); | ||
| 6191 | else | ||
| 6192 | MISSING_CASE(encoder->hpd_pin); | ||
| 6193 | |||
| 6194 | return false; | ||
| 6195 | } | 6078 | } |
| 6196 | 6079 | ||
| 6197 | /* | 6080 | /* |
| @@ -6205,44 +6088,15 @@ static bool icp_digital_port_connected(struct intel_encoder *encoder) · 7 unmodified lines | |||
| 6205 | * | 6088 | * |
| 6206 | * Return %true if port is connected, %false otherwise. | 6089 | * Return %true if port is connected, %false otherwise. |
| 6207 | */ | 6090 | */ |
| 6208 | static bool __intel_digital_port_connected(struct intel_encoder *encoder) | ||
| 6209 | { | ||
| 6210 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | ||
| 6211 | |||
| 6212 | if (HAS_GMCH(dev_priv)) { | ||
| 6213 | if (IS_GM45(dev_priv)) | ||
| 6214 | return gm45_digital_port_connected(encoder); | ||
| 6215 | else | ||
| 6216 | return g4x_digital_port_connected(encoder); | ||
| 6217 | } | ||
| 6218 | |||
| 6219 | if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP) | ||
| 6220 | return icp_digital_port_connected(encoder); | ||
| 6221 | else if (INTEL_PCH_TYPE(dev_priv) >= PCH_SPT) | ||
| 6222 | return spt_digital_port_connected(encoder); | ||
| 6223 | else if (IS_GEN9_LP(dev_priv)) | ||
| 6224 | return bxt_digital_port_connected(encoder); | ||
| 6225 | else if (IS_GEN(dev_priv, 8)) | ||
| 6226 | return bdw_digital_port_connected(encoder); | ||
| 6227 | else if (IS_GEN(dev_priv, 7)) | ||
| 6228 | return ivb_digital_port_connected(encoder); | ||
| 6229 | else if (IS_GEN(dev_priv, 6)) | ||
| 6230 | return snb_digital_port_connected(encoder); | ||
| 6231 | else if (IS_GEN(dev_priv, 5)) | ||
| 6232 | return ilk_digital_port_connected(encoder); | ||
| 6233 | |||
| 6234 | MISSING_CASE(INTEL_GEN(dev_priv)); | ||
| 6235 | return false; | ||
| 6236 | } | ||
| 6237 | |||
| 6238 | bool intel_digital_port_connected(struct intel_encoder *encoder) | 6091 | bool intel_digital_port_connected(struct intel_encoder *encoder) |
| 6239 | { | 6092 | { |
| 6240 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); | 6093 | struct drm_i915_private *dev_priv = to_i915(encoder->base.dev); |
| 6094 | struct intel_digital_port *dig_port = enc_to_dig_port(encoder); | ||
| 6241 | bool is_connected = false; | 6095 | bool is_connected = false; |
| 6242 | intel_wakeref_t wakeref; | 6096 | intel_wakeref_t wakeref; |
| 6243 | 6097 | ||
| 6244 | with_intel_display_power(dev_priv, POWER_DOMAIN_DISPLAY_CORE, wakeref) | 6098 | with_intel_display_power(dev_priv, POWER_DOMAIN_DISPLAY_CORE, wakeref) |
| 6245 | is_connected = __intel_digital_port_connected(encoder); | 6099 | is_connected = dig_port->connected(encoder); |
| 6246 | 6100 | ||
| 6247 | return is_connected; | 6101 | return is_connected; |
| 6248 | } | 6102 | } |
| @@ -8522,6 +8376,18 @@ bool intel_dp_init(struct drm_i915_private *dev_priv, · 2273 unmodified lines | |||
| 8522 | 8376 | ||
| 8523 | intel_dig_port->hpd_pulse = intel_dp_hpd_pulse; | 8377 | intel_dig_port->hpd_pulse = intel_dp_hpd_pulse; |
| 8524 | 8378 | ||
| 8379 | if (HAS_GMCH(dev_priv)) { | ||
| 8380 | if (IS_GM45(dev_priv)) | ||
| 8381 | intel_dig_port->connected = gm45_digital_port_connected; | ||
| 8382 | else | ||
| 8383 | intel_dig_port->connected = g4x_digital_port_connected; | ||
| 8384 | } else { | ||
| 8385 | if (port == PORT_A) | ||
| 8386 | intel_dig_port->connected = ilk_digital_port_connected; | ||
| 8387 | else | ||
| 8388 | intel_dig_port->connected = ibx_digital_port_connected; | ||
| 8389 | } | ||
| 8390 | |||
| 8525 | if (port != PORT_A) | 8391 | if (port != PORT_A) |
| 8526 | intel_infoframe_init(intel_dig_port); | 8392 | intel_infoframe_init(intel_dig_port); |
| 8527 | 8393 | ||
| @@ -16,6 +16,7 @@ struct drm_connector_state; · 15 unmodified lines | |||
| 16 | struct drm_encoder; | 16 | struct drm_encoder; |
| 17 | struct drm_i915_private; | 17 | struct drm_i915_private; |
| 18 | struct drm_modeset_acquire_ctx; | 18 | struct drm_modeset_acquire_ctx; |
| 19 | struct drm_dp_vsc_sdp; | ||
| 19 | struct intel_connector; | 20 | struct intel_connector; |
| 20 | struct intel_crtc_state; | 21 | struct intel_crtc_state; |
| 21 | struct intel_digital_port; | 22 | struct intel_digital_port; |
| @@ -108,15 +109,19 @@ int intel_dp_link_required(int pixel_clock, int bpp); · 86 unmodified lines | |||
| 108 | int intel_dp_max_data_rate(int max_link_clock, int max_lanes); | 109 | int intel_dp_max_data_rate(int max_link_clock, int max_lanes); |
| 109 | bool intel_dp_needs_vsc_sdp(const struct intel_crtc_state *crtc_state, | 110 | bool intel_dp_needs_vsc_sdp(const struct intel_crtc_state *crtc_state, |
| 110 | const struct drm_connector_state *conn_state); | 111 | const struct drm_connector_state *conn_state); |
| 111 | void intel_dp_vsc_enable(struct intel_dp *intel_dp, | 112 | void intel_dp_compute_psr_vsc_sdp(struct intel_dp *intel_dp, |
| 112 | const struct intel_crtc_state *crtc_state, | ||
| 113 | const struct drm_connector_state *conn_state); | ||
| 114 | void intel_dp_hdr_metadata_enable(struct intel_dp *intel_dp, | ||
| 115 | const struct intel_crtc_state *crtc_state, | 113 | const struct intel_crtc_state *crtc_state, |
| 116 | const struct drm_connector_state *conn_state); | 114 | const struct drm_connector_state *conn_state, |
| 115 | struct drm_dp_vsc_sdp *vsc); | ||
| 116 | void intel_write_dp_vsc_sdp(struct intel_encoder *encoder, | ||
| 117 | const struct intel_crtc_state *crtc_state, | ||
| 118 | struct drm_dp_vsc_sdp *vsc); | ||
| 117 | void intel_dp_set_infoframes(struct intel_encoder *encoder, bool enable, | 119 | void intel_dp_set_infoframes(struct intel_encoder *encoder, bool enable, |
| 118 | const struct intel_crtc_state *crtc_state, | 120 | const struct intel_crtc_state *crtc_state, |
| 119 | const struct drm_connector_state *conn_state); | 121 | const struct drm_connector_state *conn_state); |
| 122 | void intel_read_dp_sdp(struct intel_encoder *encoder, | ||
| 123 | struct intel_crtc_state *crtc_state, | ||
| 124 | unsigned int type); | ||
| 120 | bool intel_digital_port_connected(struct intel_encoder *encoder); | 125 | bool intel_digital_port_connected(struct intel_encoder *encoder); |
| 121 | void intel_dp_process_phy_request(struct intel_dp *intel_dp); | 126 | void intel_dp_process_phy_request(struct intel_dp *intel_dp); |
| 122 | 127 | ||
| @@ -510,10 +510,6 @@ static void intel_mst_enable_dp(struct intel_atomic_state *state, · 509 unmodified lines | |||
| 510 | 510 | ||
| 511 | intel_ddi_enable_transcoder_func(encoder, pipe_config); | 511 | intel_ddi_enable_transcoder_func(encoder, pipe_config); |
| 512 | 512 | ||
| 513 | intel_enable_pipe(pipe_config); | ||
| 514 | |||
| 515 | intel_crtc_vblank_on(pipe_config); | ||
| 516 | |||
| 517 | drm_dbg_kms(&dev_priv->drm, "active links %d\n", | 513 | drm_dbg_kms(&dev_priv->drm, "active links %d\n", |
| 518 | intel_dp->active_mst_links); | 514 | intel_dp->active_mst_links); |
| 519 | 515 | ||
| @@ -524,6 +520,11 @@ static void intel_mst_enable_dp(struct intel_atomic_state *state, · 4 unmodified lines | |||
| 524 | drm_dp_check_act_status(&intel_dp->mst_mgr); | 520 | drm_dp_check_act_status(&intel_dp->mst_mgr); |
| 525 | 521 | ||
| 526 | drm_dp_update_payload_part2(&intel_dp->mst_mgr); | 522 | drm_dp_update_payload_part2(&intel_dp->mst_mgr); |
| 523 | |||
| 524 | intel_enable_pipe(pipe_config); | ||
| 525 | |||
| 526 | intel_crtc_vblank_on(pipe_config); | ||
| 527 | |||
| 527 | if (pipe_config->has_audio) | 528 | if (pipe_config->has_audio) |
| 528 | intel_audio_codec_enable(encoder, pipe_config, conn_state); | 529 | intel_audio_codec_enable(encoder, pipe_config, conn_state); |
| 529 | } | 530 | } |
| @@ -540,6 +540,9 @@ static void __intel_fbc_cleanup_cfb(struct drm_i915_private *dev_priv) · 539 unmodified lines | |||
| 540 | { | 540 | { |
| 541 | struct intel_fbc *fbc = &dev_priv->fbc; | 541 | struct intel_fbc *fbc = &dev_priv->fbc; |
| 542 | 542 | ||
| 543 | if (WARN_ON(intel_fbc_hw_is_active(dev_priv))) | ||
| 544 | return; | ||
| 545 | |||
| 543 | if (!drm_mm_node_allocated(&fbc->compressed_fb)) | 546 | if (!drm_mm_node_allocated(&fbc->compressed_fb)) |
| 544 | return; | 547 | return; |
| 545 | 548 | ||
| @@ -564,7 +567,7 @@ void intel_fbc_cleanup_cfb(struct drm_i915_private *dev_priv) · 18 unmodified lines | |||
| 564 | } | 567 | } |
| 565 | 568 | ||
| 566 | static bool stride_is_valid(struct drm_i915_private *dev_priv, | 569 | static bool stride_is_valid(struct drm_i915_private *dev_priv, |
| 567 | unsigned int stride) | 570 | u64 modifier, unsigned int stride) |
| 568 | { | 571 | { |
| 569 | /* This should have been caught earlier. */ | 572 | /* This should have been caught earlier. */ |
| 570 | if (drm_WARN_ON_ONCE(&dev_priv->drm, (stride & (64 - 1)) != 0)) | 573 | if (drm_WARN_ON_ONCE(&dev_priv->drm, (stride & (64 - 1)) != 0)) |
| @@ -580,6 +583,11 @@ static bool stride_is_valid(struct drm_i915_private *dev_priv, · 9 unmodified lines | |||
| 580 | if (IS_GEN(dev_priv, 4) && !IS_G4X(dev_priv) && stride < 2048) | 583 | if (IS_GEN(dev_priv, 4) && !IS_G4X(dev_priv) && stride < 2048) |
| 581 | return false; | 584 | return false; |
| 582 | 585 | ||
| 586 | /* Display WA #1105: skl,bxt,kbl,cfl,glk */ | ||
| 587 | if (IS_GEN(dev_priv, 9) && | ||
| 588 | modifier == DRM_FORMAT_MOD_LINEAR && stride & 511) | ||
| 589 | return false; | ||
| 590 | |||
| 583 | if (stride > 16384) | 591 | if (stride > 16384) |
| 584 | return false; | 592 | return false; |
| 585 | 593 | ||
| @@ -810,7 +818,7 @@ static bool intel_fbc_can_activate(struct intel_crtc *crtc) · 224 unmodified lines | |||
| 810 | return false; | 818 | return false; |
| 811 | } | 819 | } |
| 812 | 820 | ||
| 813 | if (!stride_is_valid(dev_priv, cache->fb.stride)) { | 821 | if (!stride_is_valid(dev_priv, cache->fb.modifier, cache->fb.stride)) { |
| 814 | fbc->no_fbc_reason = "framebuffer stride not supported"; | 822 | fbc->no_fbc_reason = "framebuffer stride not supported"; |
| 815 | return false; | 823 | return false; |
| 816 | } | 824 | } |
| @@ -522,7 +522,7 @@ u32 lspcon_infoframes_enabled(struct intel_encoder *encoder, · 521 unmodified lines | |||
| 522 | const struct intel_crtc_state *pipe_config) | 522 | const struct intel_crtc_state *pipe_config) |
| 523 | { | 523 | { |
| 524 | /* FIXME actually read this from the hw */ | 524 | /* FIXME actually read this from the hw */ |
| 525 | return enc_to_intel_lspcon(encoder)->active; | 525 | return 0; |
| 526 | } | 526 | } |
| 527 | 527 | ||
| 528 | void lspcon_resume(struct intel_lspcon *lspcon) | 528 | void lspcon_resume(struct intel_lspcon *lspcon) |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | |||
| 30 | #include "intel_display_types.h" | 30 | #include "intel_display_types.h" |
| 31 | #include "intel_psr.h" | 31 | #include "intel_psr.h" |
| 32 | #include "intel_sprite.h" | 32 | #include "intel_sprite.h" |
| 33 | #include "intel_hdmi.h" | ||
| 33 | 34 | ||
| 34 | /** | 35 | /** |
| 35 | * DOC: Panel Self Refresh (PSR/SRD) | 36 | * DOC: Panel Self Refresh (PSR/SRD) |
| @@ -357,39 +358,6 @@ void intel_psr_init_dpcd(struct intel_dp *intel_dp) · 321 unmodified lines | |||
| 357 | } | 358 | } |
| 358 | } | 359 | } |
| 359 | 360 | ||
| 360 | static void intel_psr_setup_vsc(struct intel_dp *intel_dp, | ||
| 361 | const struct intel_crtc_state *crtc_state) | ||
| 362 | { | ||
| 363 | struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp); | ||
| 364 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); | ||
| 365 | struct dp_sdp psr_vsc; | ||
| 366 | |||
| 367 | if (dev_priv->psr.psr2_enabled) { | ||
| 368 | /* Prepare VSC Header for SU as per EDP 1.4 spec, Table 6.11 */ | ||
| 369 | memset(&psr_vsc, 0, sizeof(psr_vsc)); | ||
| 370 | psr_vsc.sdp_header.HB0 = 0; | ||
| 371 | psr_vsc.sdp_header.HB1 = 0x7; | ||
| 372 | if (dev_priv->psr.colorimetry_support) { | ||
| 373 | psr_vsc.sdp_header.HB2 = 0x5; | ||
| 374 | psr_vsc.sdp_header.HB3 = 0x13; | ||
| 375 | } else { | ||
| 376 | psr_vsc.sdp_header.HB2 = 0x4; | ||
| 377 | psr_vsc.sdp_header.HB3 = 0xe; | ||
| 378 | } | ||
| 379 | } else { | ||
| 380 | /* Prepare VSC packet as per EDP 1.3 spec, Table 3.10 */ | ||
| 381 | memset(&psr_vsc, 0, sizeof(psr_vsc)); | ||
| 382 | psr_vsc.sdp_header.HB0 = 0; | ||
| 383 | psr_vsc.sdp_header.HB1 = 0x7; | ||
| 384 | psr_vsc.sdp_header.HB2 = 0x2; | ||
| 385 | psr_vsc.sdp_header.HB3 = 0x8; | ||
| 386 | } | ||
| 387 | |||
| 388 | intel_dig_port->write_infoframe(&intel_dig_port->base, | ||
| 389 | crtc_state, | ||
| 390 | DP_SDP_VSC, &psr_vsc, sizeof(psr_vsc)); | ||
| 391 | } | ||
| 392 | |||
| 393 | static void hsw_psr_setup_aux(struct intel_dp *intel_dp) | 361 | static void hsw_psr_setup_aux(struct intel_dp *intel_dp) |
| 394 | { | 362 | { |
| 395 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); | 363 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); |
| @@ -756,6 +724,8 @@ void intel_psr_compute_config(struct intel_dp *intel_dp, · 360 unmodified lines | |||
| 756 | if (intel_dp != dev_priv->psr.dp) | 724 | if (intel_dp != dev_priv->psr.dp) |
| 757 | return; | 725 | return; |
| 758 | 726 | ||
| 727 | if (!psr_global_enabled(dev_priv)) | ||
| 728 | return; | ||
| 759 | /* | 729 | /* |
| 760 | * HSW spec explicitly says PSR is tied to port A. | 730 | * HSW spec explicitly says PSR is tied to port A. |
| 761 | * BDW+ platforms have a instance of PSR registers per transcoder but | 731 | * BDW+ platforms have a instance of PSR registers per transcoder but |
| @@ -798,6 +768,7 @@ void intel_psr_compute_config(struct intel_dp *intel_dp, · 36 unmodified lines | |||
| 798 | 768 | ||
| 799 | crtc_state->has_psr = true; | 769 | crtc_state->has_psr = true; |
| 800 | crtc_state->has_psr2 = intel_psr2_config_valid(intel_dp, crtc_state); | 770 | crtc_state->has_psr2 = intel_psr2_config_valid(intel_dp, crtc_state); |
| 771 | crtc_state->infoframes.enable |= intel_hdmi_infoframe_enable(DP_SDP_VSC); | ||
| 801 | } | 772 | } |
| 802 | 773 | ||
| 803 | static void intel_psr_activate(struct intel_dp *intel_dp) | 774 | static void intel_psr_activate(struct intel_dp *intel_dp) |
| @@ -880,9 +851,12 @@ static void intel_psr_enable_source(struct intel_dp *intel_dp, · 76 unmodified lines | |||
| 880 | } | 851 | } |
| 881 | 852 | ||
| 882 | static void intel_psr_enable_locked(struct drm_i915_private *dev_priv, | 853 | static void intel_psr_enable_locked(struct drm_i915_private *dev_priv, |
| 883 | const struct intel_crtc_state *crtc_state) | 854 | const struct intel_crtc_state *crtc_state, |
| 855 | const struct drm_connector_state *conn_state) | ||
| 884 | { | 856 | { |
| 885 | struct intel_dp *intel_dp = dev_priv->psr.dp; | 857 | struct intel_dp *intel_dp = dev_priv->psr.dp; |
| 858 | struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp); | ||
| 859 | struct intel_encoder *encoder = &intel_dig_port->base; | ||
| 886 | u32 val; | 860 | u32 val; |
| 887 | 861 | ||
| 888 | drm_WARN_ON(&dev_priv->drm, dev_priv->psr.enabled); | 862 | drm_WARN_ON(&dev_priv->drm, dev_priv->psr.enabled); |
| @@ -921,7 +895,9 @@ static void intel_psr_enable_locked(struct drm_i915_private *dev_priv, · 32 unmodified lines | |||
| 921 | 895 | ||
| 922 | drm_dbg_kms(&dev_priv->drm, "Enabling PSR%s\n", | 896 | drm_dbg_kms(&dev_priv->drm, "Enabling PSR%s\n", |
| 923 | dev_priv->psr.psr2_enabled ? "2" : "1"); | 897 | dev_priv->psr.psr2_enabled ? "2" : "1"); |
| 924 | intel_psr_setup_vsc(intel_dp, crtc_state); | 898 | intel_dp_compute_psr_vsc_sdp(intel_dp, crtc_state, conn_state, |
| 899 | &dev_priv->psr.vsc); | ||
| 900 | intel_write_dp_vsc_sdp(encoder, crtc_state, &dev_priv->psr.vsc); | ||
| 925 | intel_psr_enable_sink(intel_dp); | 901 | intel_psr_enable_sink(intel_dp); |
| 926 | intel_psr_enable_source(intel_dp, crtc_state); | 902 | intel_psr_enable_source(intel_dp, crtc_state); |
| 927 | dev_priv->psr.enabled = true; | 903 | dev_priv->psr.enabled = true; |
| @@ -933,11 +909,13 @@ static void intel_psr_enable_locked(struct drm_i915_private *dev_priv, · 5 unmodified lines | |||
| 933 | * intel_psr_enable - Enable PSR | 909 | * intel_psr_enable - Enable PSR |
| 934 | * @intel_dp: Intel DP | 910 | * @intel_dp: Intel DP |
| 935 | * @crtc_state: new CRTC state | 911 | * @crtc_state: new CRTC state |
| 912 | * @conn_state: new CONNECTOR state | ||
| 936 | * | 913 | * |
| 937 | * This function can only be called after the pipe is fully trained and enabled. | 914 | * This function can only be called after the pipe is fully trained and enabled. |
| 938 | */ | 915 | */ |
| 939 | void intel_psr_enable(struct intel_dp *intel_dp, | 916 | void intel_psr_enable(struct intel_dp *intel_dp, |
| 940 | const struct intel_crtc_state *crtc_state) | 917 | const struct intel_crtc_state *crtc_state, |
| 918 | const struct drm_connector_state *conn_state) | ||
| 941 | { | 919 | { |
| 942 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); | 920 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); |
| 943 | 921 | ||
| @@ -958,7 +936,7 @@ void intel_psr_enable(struct intel_dp *intel_dp, · 14 unmodified lines | |||
| 958 | goto unlock; | 936 | goto unlock; |
| 959 | } | 937 | } |
| 960 | 938 | ||
| 961 | intel_psr_enable_locked(dev_priv, crtc_state); | 939 | intel_psr_enable_locked(dev_priv, crtc_state, conn_state); |
| 962 | 940 | ||
| 963 | unlock: | 941 | unlock: |
| 964 | mutex_unlock(&dev_priv->psr.lock); | 942 | mutex_unlock(&dev_priv->psr.lock); |
| @@ -1091,13 +1069,15 @@ static void psr_force_hw_tracking_exit(struct drm_i915_private *dev_priv) · 126 unmodified lines | |||
| 1091 | * intel_psr_update - Update PSR state | 1069 | * intel_psr_update - Update PSR state |
| 1092 | * @intel_dp: Intel DP | 1070 | * @intel_dp: Intel DP |
| 1093 | * @crtc_state: new CRTC state | 1071 | * @crtc_state: new CRTC state |
| 1072 | * @conn_state: new CONNECTOR state | ||
| 1094 | * | 1073 | * |
| 1095 | * This functions will update PSR states, disabling, enabling or switching PSR | 1074 | * This functions will update PSR states, disabling, enabling or switching PSR |
| 1096 | * version when executing fastsets. For full modeset, intel_psr_disable() and | 1075 | * version when executing fastsets. For full modeset, intel_psr_disable() and |
| 1097 | * intel_psr_enable() should be called instead. | 1076 | * intel_psr_enable() should be called instead. |
| 1098 | */ | 1077 | */ |
| 1099 | void intel_psr_update(struct intel_dp *intel_dp, | 1078 | void intel_psr_update(struct intel_dp *intel_dp, |
| 1100 | const struct intel_crtc_state *crtc_state) | 1079 | const struct intel_crtc_state *crtc_state, |
| 1080 | const struct drm_connector_state *conn_state) | ||
| 1101 | { | 1081 | { |
| 1102 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); | 1082 | struct drm_i915_private *dev_priv = dp_to_i915(intel_dp); |
| 1103 | struct i915_psr *psr = &dev_priv->psr; | 1083 | struct i915_psr *psr = &dev_priv->psr; |
| @@ -1134,7 +1114,7 @@ void intel_psr_update(struct intel_dp *intel_dp, · 30 unmodified lines | |||
| 1134 | intel_psr_disable_locked(intel_dp); | 1114 | intel_psr_disable_locked(intel_dp); |
| 1135 | 1115 | ||
| 1136 | if (enable) | 1116 | if (enable) |
| 1137 | intel_psr_enable_locked(dev_priv, crtc_state); | 1117 | intel_psr_enable_locked(dev_priv, crtc_state, conn_state); |
| 1138 | 1118 | ||
| 1139 | unlock: | 1119 | unlock: |
| 1140 | mutex_unlock(&dev_priv->psr.lock); | 1120 | mutex_unlock(&dev_priv->psr.lock); |
| @@ -17,11 +17,13 @@ struct intel_dp; · 16 unmodified lines | |||
| 17 | #define CAN_PSR(dev_priv) (HAS_PSR(dev_priv) && dev_priv->psr.sink_support) | 17 | #define CAN_PSR(dev_priv) (HAS_PSR(dev_priv) && dev_priv->psr.sink_support) |
| 18 | void intel_psr_init_dpcd(struct intel_dp *intel_dp); | 18 | void intel_psr_init_dpcd(struct intel_dp *intel_dp); |
| 19 | void intel_psr_enable(struct intel_dp *intel_dp, | 19 | void intel_psr_enable(struct intel_dp *intel_dp, |
| 20 | const struct intel_crtc_state *crtc_state); | 20 | const struct intel_crtc_state *crtc_state, |
| 21 | const struct drm_connector_state *conn_state); | ||
| 21 | void intel_psr_disable(struct intel_dp *intel_dp, | 22 | void intel_psr_disable(struct intel_dp *intel_dp, |
| 22 | const struct intel_crtc_state *old_crtc_state); | 23 | const struct intel_crtc_state *old_crtc_state); |
| 23 | void intel_psr_update(struct intel_dp *intel_dp, | 24 | void intel_psr_update(struct intel_dp *intel_dp, |
| 24 | const struct intel_crtc_state *crtc_state); | 25 | const struct intel_crtc_state *crtc_state, |
| 26 | const struct drm_connector_state *conn_state); | ||
| 25 | int intel_psr_debug_set(struct drm_i915_private *dev_priv, u64 value); | 27 | int intel_psr_debug_set(struct drm_i915_private *dev_priv, u64 value); |
| 26 | void intel_psr_invalidate(struct drm_i915_private *dev_priv, | 28 | void intel_psr_invalidate(struct drm_i915_private *dev_priv, |
| 27 | unsigned frontbuffer_bits, | 29 | unsigned frontbuffer_bits, |
| @@ -238,8 +238,8 @@ static void tc_port_fixup_legacy_flag(struct intel_digital_port *dig_port, · 237 unmodified lines | |||
| 238 | static u32 tc_port_live_status_mask(struct intel_digital_port *dig_port) | 238 | static u32 tc_port_live_status_mask(struct intel_digital_port *dig_port) |
| 239 | { | 239 | { |
| 240 | struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); | 240 | struct drm_i915_private *i915 = to_i915(dig_port->base.base.dev); |
| 241 | enum tc_port tc_port = intel_port_to_tc(i915, dig_port->base.port); | ||
| 242 | struct intel_uncore *uncore = &i915->uncore; | 241 | struct intel_uncore *uncore = &i915->uncore; |
| 242 | u32 isr_bit = i915->hotplug.pch_hpd[dig_port->base.hpd_pin]; | ||
| 243 | u32 mask = 0; | 243 | u32 mask = 0; |
| 244 | u32 val; | 244 | u32 val; |
| 245 | 245 | ||
| @@ -258,7 +258,7 @@ static u32 tc_port_live_status_mask(struct intel_digital_port *dig_port) · 12 unmodified lines | |||
| 258 | if (val & TC_LIVE_STATE_TC(dig_port->tc_phy_fia_idx)) | 258 | if (val & TC_LIVE_STATE_TC(dig_port->tc_phy_fia_idx)) |
| 259 | mask |= BIT(TC_PORT_DP_ALT); | 259 | mask |= BIT(TC_PORT_DP_ALT); |
| 260 | 260 | ||
| 261 | if (intel_uncore_read(uncore, SDEISR) & SDE_TC_HOTPLUG_ICP(tc_port)) | 261 | if (intel_uncore_read(uncore, SDEISR) & isr_bit) |
| 262 | mask |= BIT(TC_PORT_LEGACY); | 262 | mask |= BIT(TC_PORT_LEGACY); |
| 263 | 263 | ||
| 264 | /* The sink can be connected only in a single mode. */ | 264 | /* The sink can be connected only in a single mode. */ |
| @@ -562,8 +562,9 @@ static bool intel_tc_port_needs_reset(struct intel_digital_port *dig_port) · 297 unmodified lines | |||
| 562 | * connected ports are usable, and avoids exposing to the users objects they | 562 | * connected ports are usable, and avoids exposing to the users objects they |
| 563 | * can't really use. | 563 | * can't really use. |
| 564 | */ | 564 | */ |
| 565 | bool intel_tc_port_connected(struct intel_digital_port *dig_port) | 565 | bool intel_tc_port_connected(struct intel_encoder *encoder) |
| 566 | { | 566 | { |
| 567 | struct intel_digital_port *dig_port = enc_to_dig_port(encoder); | ||
| 567 | bool is_connected; | 568 | bool is_connected; |
| 568 | intel_wakeref_t tc_cold_wref; | 569 | intel_wakeref_t tc_cold_wref; |
| 569 | 570 | ||
| @@ -10,8 +10,9 @@ · 9 unmodified lines | |||
| 10 | #include <linux/types.h> | 10 | #include <linux/types.h> |
| 11 | 11 | ||
| 12 | struct intel_digital_port; | 12 | struct intel_digital_port; |
| 13 | struct intel_encoder; | ||
| 13 | 14 | ||
| 14 | bool intel_tc_port_connected(struct intel_digital_port *dig_port); | 15 | bool intel_tc_port_connected(struct intel_encoder *encoder); |
| 15 | u32 intel_tc_port_get_lane_mask(struct intel_digital_port *dig_port); | 16 | u32 intel_tc_port_get_lane_mask(struct intel_digital_port *dig_port); |
| 16 | u32 intel_tc_port_get_pin_assignment_mask(struct intel_digital_port *dig_port); | 17 | u32 intel_tc_port_get_pin_assignment_mask(struct intel_digital_port *dig_port); |
| 17 | int intel_tc_port_fia_max_lane_count(struct intel_digital_port *dig_port); | 18 | int intel_tc_port_fia_max_lane_count(struct intel_digital_port *dig_port); |
| @@ -462,7 +462,7 @@ struct bdb_general_definitions { · 461 unmodified lines | |||
| 462 | * number = (block_size - sizeof(bdb_general_definitions))/ | 462 | * number = (block_size - sizeof(bdb_general_definitions))/ |
| 463 | * defs->child_dev_size; | 463 | * defs->child_dev_size; |
| 464 | */ | 464 | */ |
| 465 | u8 devices[0]; | 465 | u8 devices[]; |
| 466 | } __packed; | 466 | } __packed; |
| 467 | 467 | ||
| 468 | /* | 468 | /* |
| @@ -839,7 +839,7 @@ struct bdb_mipi_config { · 370 unmodified lines | |||
| 839 | 839 | ||
| 840 | struct bdb_mipi_sequence { | 840 | struct bdb_mipi_sequence { |
| 841 | u8 version; | 841 | u8 version; |
| 842 | u8 data[0]; /* up to 6 variable length blocks */ | 842 | u8 data[]; /* up to 6 variable length blocks */ |
| 843 | } __packed; | 843 | } __packed; |
| 844 | 844 | ||
| 845 | /* | 845 | /* |
| @@ -108,7 +108,7 @@ bool i915_gem_clflush_object(struct drm_i915_gem_object *obj, · 107 unmodified lines | |||
| 108 | if (clflush) { | 108 | if (clflush) { |
| 109 | i915_sw_fence_await_reservation(&clflush->base.chain, | 109 | i915_sw_fence_await_reservation(&clflush->base.chain, |
| 110 | obj->base.resv, NULL, true, | 110 | obj->base.resv, NULL, true, |
| 111 | I915_FENCE_TIMEOUT, | 111 | i915_fence_timeout(to_i915(obj->base.dev)), |
| 112 | I915_FENCE_GFP); | 112 | I915_FENCE_GFP); |
| 113 | dma_resv_add_excl_fence(obj->base.resv, &clflush->base.dma); | 113 | dma_resv_add_excl_fence(obj->base.resv, &clflush->base.dma); |
| 114 | dma_fence_work_commit(&clflush->base); | 114 | dma_fence_work_commit(&clflush->base); |
| @@ -6,7 +6,6 @@ · 5 unmodified lines | |||
| 6 | #include "i915_drv.h" | 6 | #include "i915_drv.h" |
| 7 | #include "gt/intel_context.h" | 7 | #include "gt/intel_context.h" |
| 8 | #include "gt/intel_engine_pm.h" | 8 | #include "gt/intel_engine_pm.h" |
| 9 | #include "gt/intel_engine_pool.h" | ||
| 10 | #include "i915_gem_client_blt.h" | 9 | #include "i915_gem_client_blt.h" |
| 11 | #include "i915_gem_object_blt.h" | 10 | #include "i915_gem_object_blt.h" |
| 12 | 11 | ||
| @@ -289,8 +288,7 @@ int i915_gem_schedule_fill_pages_blt(struct drm_i915_gem_object *obj, · 276 unmodified lines | |||
| 289 | 288 | ||
| 290 | i915_gem_object_lock(obj); | 289 | i915_gem_object_lock(obj); |
| 291 | err = i915_sw_fence_await_reservation(&work->wait, | 290 | err = i915_sw_fence_await_reservation(&work->wait, |
| 292 | obj->base.resv, NULL, | 291 | obj->base.resv, NULL, true, 0, |
| 293 | true, I915_FENCE_TIMEOUT, | ||
| 294 | I915_FENCE_GFP); | 292 | I915_FENCE_GFP); |
| 295 | if (err < 0) { | 293 | if (err < 0) { |
| 296 | dma_fence_set_error(&work->dma, err); | 294 | dma_fence_set_error(&work->dma, err); |
| @@ -15,8 +15,8 @@ · 14 unmodified lines | |||
| 15 | 15 | ||
| 16 | #include "gem/i915_gem_ioctls.h" | 16 | #include "gem/i915_gem_ioctls.h" |
| 17 | #include "gt/intel_context.h" | 17 | #include "gt/intel_context.h" |
| 18 | #include "gt/intel_engine_pool.h" | ||
| 19 | #include "gt/intel_gt.h" | 18 | #include "gt/intel_gt.h" |
| 19 | #include "gt/intel_gt_buffer_pool.h" | ||
| 20 | #include "gt/intel_gt_pm.h" | 20 | #include "gt/intel_gt_pm.h" |
| 21 | #include "gt/intel_ring.h" | 21 | #include "gt/intel_ring.h" |
| 22 | 22 | ||
| @@ -268,7 +268,9 @@ struct i915_execbuffer { · 245 unmodified lines | |||
| 268 | bool has_fence : 1; | 268 | bool has_fence : 1; |
| 269 | bool needs_unfenced : 1; | 269 | bool needs_unfenced : 1; |
| 270 | 270 | ||
| 271 | struct i915_vma *target; | ||
| 271 | struct i915_request *rq; | 272 | struct i915_request *rq; |
| 273 | struct i915_vma *rq_vma; | ||
| 272 | u32 *rq_cmd; | 274 | u32 *rq_cmd; |
| 273 | unsigned int rq_size; | 275 | unsigned int rq_size; |
| 274 | } reloc_cache; | 276 | } reloc_cache; |
| @@ -953,7 +955,7 @@ static void reloc_cache_init(struct reloc_cache *cache, · 678 unmodified lines | |||
| 953 | cache->needs_unfenced = INTEL_INFO(i915)->unfenced_needs_alignment; | 955 | cache->needs_unfenced = INTEL_INFO(i915)->unfenced_needs_alignment; |
| 954 | cache->node.flags = 0; | 956 | cache->node.flags = 0; |
| 955 | cache->rq = NULL; | 957 | cache->rq = NULL; |
| 956 | cache->rq_size = 0; | 958 | cache->target = NULL; |
| 957 | } | 959 | } |
| 958 | 960 | ||
| 959 | static inline void *unmask_page(unsigned long p) | 961 | static inline void *unmask_page(unsigned long p) |
| @@ -975,29 +977,122 @@ static inline struct i915_ggtt *cache_to_ggtt(struct reloc_cache *cache) · 15 unmodified lines | |||
| 975 | return &i915->ggtt; | 977 | return &i915->ggtt; |
| 976 | } | 978 | } |
| 977 | 979 | ||
| 978 | static void reloc_gpu_flush(struct reloc_cache *cache) | 980 | #define RELOC_TAIL 4 |
| 981 | |||
| 982 | static int reloc_gpu_chain(struct reloc_cache *cache) | ||
| 979 | { | 983 | { |
| 980 | struct drm_i915_gem_object *obj = cache->rq->batch->obj; | 984 | struct intel_gt_buffer_pool_node *pool; |
| 985 | struct i915_request *rq = cache->rq; | ||
| 986 | struct i915_vma *batch; | ||
| 987 | u32 *cmd; | ||
| 988 | int err; | ||
| 981 | 989 | ||
| 982 | GEM_BUG_ON(cache->rq_size >= obj->base.size / sizeof(u32)); | 990 | pool = intel_gt_get_buffer_pool(rq->engine->gt, PAGE_SIZE); |
| 983 | cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END; | 991 | if (IS_ERR(pool)) |
| 992 | return PTR_ERR(pool); | ||
| 984 | 993 | ||
| 985 | __i915_gem_object_flush_map(obj, 0, sizeof(u32) * (cache->rq_size + 1)); | 994 | batch = i915_vma_instance(pool->obj, rq->context->vm, NULL); |
| 986 | i915_gem_object_unpin_map(obj); | 995 | if (IS_ERR(batch)) { |
| 996 | err = PTR_ERR(batch); | ||
| 997 | goto out_pool; | ||
| 998 | } | ||
| 987 | 999 | ||
| 988 | intel_gt_chipset_flush(cache->rq->engine->gt); | 1000 | err = i915_vma_pin(batch, 0, 0, PIN_USER | PIN_NONBLOCK); |
| 1001 | if (err) | ||
| 1002 | goto out_pool; | ||
| 989 | 1003 | ||
| 990 | i915_request_add(cache->rq); | 1004 | GEM_BUG_ON(cache->rq_size + RELOC_TAIL > PAGE_SIZE / sizeof(u32)); |
| 991 | cache->rq = NULL; | 1005 | cmd = cache->rq_cmd + cache->rq_size; |
| 1006 | *cmd++ = MI_ARB_CHECK; | ||
| 1007 | if (cache->gen >= 8) | ||
| 1008 | *cmd++ = MI_BATCH_BUFFER_START_GEN8; | ||
| 1009 | else if (cache->gen >= 6) | ||
| 1010 | *cmd++ = MI_BATCH_BUFFER_START; | ||
| 1011 | else | ||
| 1012 | *cmd++ = MI_BATCH_BUFFER_START | MI_BATCH_GTT; | ||
| 1013 | *cmd++ = lower_32_bits(batch->node.start); | ||
| 1014 | *cmd++ = upper_32_bits(batch->node.start); /* Always 0 for gen<8 */ | ||
| 1015 | i915_gem_object_flush_map(cache->rq_vma->obj); | ||
| 1016 | i915_gem_object_unpin_map(cache->rq_vma->obj); | ||
| 1017 | cache->rq_vma = NULL; | ||
| 1018 | |||
| 1019 | err = intel_gt_buffer_pool_mark_active(pool, rq); | ||
| 1020 | if (err == 0) { | ||
| 1021 | i915_vma_lock(batch); | ||
| 1022 | err = i915_request_await_object(rq, batch->obj, false); | ||
| 1023 | if (err == 0) | ||
| 1024 | err = i915_vma_move_to_active(batch, rq, 0); | ||
| 1025 | i915_vma_unlock(batch); | ||
| 1026 | } | ||
| 1027 | i915_vma_unpin(batch); | ||
| 1028 | if (err) | ||
| 1029 | goto out_pool; | ||
| 1030 | |||
| 1031 | cmd = i915_gem_object_pin_map(batch->obj, | ||
| 1032 | cache->has_llc ? | ||
| 1033 | I915_MAP_FORCE_WB : | ||
| 1034 | I915_MAP_FORCE_WC); | ||
| 1035 | if (IS_ERR(cmd)) { | ||
| 1036 | err = PTR_ERR(cmd); | ||
| 1037 | goto out_pool; | ||
| 1038 | } | ||
| 1039 | |||
| 1040 | /* Return with batch mapping (cmd) still pinned */ | ||
| 1041 | cache->rq_cmd = cmd; | ||
| 1042 | cache->rq_size = 0; | ||
| 1043 | cache->rq_vma = batch; | ||
| 1044 | |||
| 1045 | out_pool: | ||
| 1046 | intel_gt_buffer_pool_put(pool); | ||
| 1047 | return err; | ||
| 1048 | } | ||
| 1049 | |||
| 1050 | static unsigned int reloc_bb_flags(const struct reloc_cache *cache) | ||
| 1051 | { | ||
| 1052 | return cache->gen > 5 ? 0 : I915_DISPATCH_SECURE; | ||
| 1053 | } | ||
| 1054 | |||
| 1055 | static int reloc_gpu_flush(struct reloc_cache *cache) | ||
| 1056 | { | ||
| 1057 | struct i915_request *rq; | ||
| 1058 | int err; | ||
| 1059 | |||
| 1060 | rq = fetch_and_zero(&cache->rq); | ||
| 1061 | if (!rq) | ||
| 1062 | return 0; | ||
| 1063 | |||
| 1064 | if (cache->rq_vma) { | ||
| 1065 | struct drm_i915_gem_object *obj = cache->rq_vma->obj; | ||
| 1066 | |||
| 1067 | GEM_BUG_ON(cache->rq_size >= obj->base.size / sizeof(u32)); | ||
| 1068 | cache->rq_cmd[cache->rq_size++] = MI_BATCH_BUFFER_END; | ||
| 1069 | |||
| 1070 | __i915_gem_object_flush_map(obj, | ||
| 1071 | 0, sizeof(u32) * cache->rq_size); | ||
| 1072 | i915_gem_object_unpin_map(obj); | ||
| 1073 | } | ||
| 1074 | |||
| 1075 | err = 0; | ||
| 1076 | if (rq->engine->emit_init_breadcrumb) | ||
| 1077 | err = rq->engine->emit_init_breadcrumb(rq); | ||
| 1078 | if (!err) | ||
| 1079 | err = rq->engine->emit_bb_start(rq, | ||
| 1080 | rq->batch->node.start, | ||
| 1081 | PAGE_SIZE, | ||
| 1082 | reloc_bb_flags(cache)); | ||
| 1083 | if (err) | ||
| 1084 | i915_request_set_error_once(rq, err); | ||
| 1085 | |||
| 1086 | intel_gt_chipset_flush(rq->engine->gt); | ||
| 1087 | i915_request_add(rq); | ||
| 1088 | |||
| 1089 | return err; | ||
| 992 | } | 1090 | } |
| 993 | 1091 | ||
| 994 | static void reloc_cache_reset(struct reloc_cache *cache) | 1092 | static void reloc_cache_reset(struct reloc_cache *cache) |
| 995 | { | 1093 | { |
| 996 | void *vaddr; | 1094 | void *vaddr; |
| 997 | 1095 | ||
| 998 | if (cache->rq) | ||
| 999 | reloc_gpu_flush(cache); | ||
| 1000 | |||
| 1001 | if (!cache->vaddr) | 1096 | if (!cache->vaddr) |
| 1002 | return; | 1097 | return; |
| 1003 | 1098 | ||
| @@ -1190,17 +1285,17 @@ static int reloc_move_to_gpu(struct i915_request *rq, struct i915_vma *vma) · 186 unmodified lines | |||
| 1190 | } | 1285 | } |
| 1191 | 1286 | ||
| 1192 | static int __reloc_gpu_alloc(struct i915_execbuffer *eb, | 1287 | static int __reloc_gpu_alloc(struct i915_execbuffer *eb, |
| 1193 | struct i915_vma *vma, | 1288 | struct intel_engine_cs *engine, |
| 1194 | unsigned int len) | 1289 | unsigned int len) |
| 1195 | { | 1290 | { |
| 1196 | struct reloc_cache *cache = &eb->reloc_cache; | 1291 | struct reloc_cache *cache = &eb->reloc_cache; |
| 1197 | struct intel_engine_pool_node *pool; | 1292 | struct intel_gt_buffer_pool_node *pool; |
| 1198 | struct i915_request *rq; | 1293 | struct i915_request *rq; |
| 1199 | struct i915_vma *batch; | 1294 | struct i915_vma *batch; |
| 1200 | u32 *cmd; | 1295 | u32 *cmd; |
| 1201 | int err; | 1296 | int err; |
| 1202 | 1297 | ||
| 1203 | pool = intel_engine_get_pool(eb->engine, PAGE_SIZE); | 1298 | pool = intel_gt_get_buffer_pool(engine->gt, PAGE_SIZE); |
| 1204 | if (IS_ERR(pool)) | 1299 | if (IS_ERR(pool)) |
| 1205 | return PTR_ERR(pool); | 1300 | return PTR_ERR(pool); |
| 1206 | 1301 | ||
| @@ -1213,7 +1308,7 @@ static int __reloc_gpu_alloc(struct i915_execbuffer *eb, · 6 unmodified lines | |||
| 1213 | goto out_pool; | 1308 | goto out_pool; |
| 1214 | } | 1309 | } |
| 1215 | 1310 | ||
| 1216 | batch = i915_vma_instance(pool->obj, vma->vm, NULL); | 1311 | batch = i915_vma_instance(pool->obj, eb->context->vm, NULL); |
| 1217 | if (IS_ERR(batch)) { | 1312 | if (IS_ERR(batch)) { |
| 1218 | err = PTR_ERR(batch); | 1313 | err = PTR_ERR(batch); |
| 1219 | goto err_unmap; | 1314 | goto err_unmap; |
| @@ -1223,26 +1318,32 @@ static int __reloc_gpu_alloc(struct i915_execbuffer *eb, · 3 unmodified lines | |||
| 1223 | if (err) | 1318 | if (err) |
| 1224 | goto err_unmap; | 1319 | goto err_unmap; |
| 1225 | 1320 | ||
| 1226 | rq = i915_request_create(eb->context); | 1321 | if (engine == eb->context->engine) { |
| 1322 | rq = i915_request_create(eb->context); | ||
| 1323 | } else { | ||
| 1324 | struct intel_context *ce; | ||
| 1325 | |||
| 1326 | ce = intel_context_create(engine); | ||
| 1327 | if (IS_ERR(ce)) { | ||
| 1328 | err = PTR_ERR(ce); | ||
| 1329 | goto err_unpin; | ||
| 1330 | } | ||
| 1331 | |||
| 1332 | i915_vm_put(ce->vm); | ||
| 1333 | ce->vm = i915_vm_get(eb->context->vm); | ||
| 1334 | |||
| 1335 | rq = intel_context_create_request(ce); | ||
| 1336 | intel_context_put(ce); | ||
| 1337 | } | ||
| 1227 | if (IS_ERR(rq)) { | 1338 | if (IS_ERR(rq)) { |
| 1228 | err = PTR_ERR(rq); | 1339 | err = PTR_ERR(rq); |
| 1229 | goto err_unpin; | 1340 | goto err_unpin; |
| 1230 | } | 1341 | } |
| 1231 | 1342 | ||
| 1232 | err = intel_engine_pool_mark_active(pool, rq); | 1343 | err = intel_gt_buffer_pool_mark_active(pool, rq); |
| 1233 | if (err) | ||
| 1234 | goto err_request; | ||
| 1235 | |||
| 1236 | err = reloc_move_to_gpu(rq, vma); | ||
| 1237 | if (err) | 1344 | if (err) |
| 1238 | goto err_request; | 1345 | goto err_request; |
| 1239 | 1346 | ||
| 1240 | err = eb->engine->emit_bb_start(rq, | ||
| 1241 | batch->node.start, PAGE_SIZE, | ||
| 1242 | cache->gen > 5 ? 0 : I915_DISPATCH_SECURE); | ||
| 1243 | if (err) | ||
| 1244 | goto skip_request; | ||
| 1245 | |||
| 1246 | i915_vma_lock(batch); | 1347 | i915_vma_lock(batch); |
| 1247 | err = i915_request_await_object(rq, batch->obj, false); | 1348 | err = i915_request_await_object(rq, batch->obj, false); |
| 1248 | if (err == 0) | 1349 | if (err == 0) |
| @@ -1257,6 +1358,7 @@ static int __reloc_gpu_alloc(struct i915_execbuffer *eb, · 8 unmodified lines | |||
| 1257 | cache->rq = rq; | 1358 | cache->rq = rq; |
| 1258 | cache->rq_cmd = cmd; | 1359 | cache->rq_cmd = cmd; |
| 1259 | cache->rq_size = 0; | 1360 | cache->rq_size = 0; |
| 1361 | cache->rq_vma = batch; | ||
| 1260 | 1362 | ||
| 1261 | /* Return with batch mapping (cmd) still pinned */ | 1363 | /* Return with batch mapping (cmd) still pinned */ |
| 1262 | goto out_pool; | 1364 | goto out_pool; |
| @@ -1270,31 +1372,57 @@ err_unpin: · 7 unmodified lines | |||
| 1270 | err_unmap: | 1372 | err_unmap: |
| 1271 | i915_gem_object_unpin_map(pool->obj); | 1373 | i915_gem_object_unpin_map(pool->obj); |
| 1272 | out_pool: | 1374 | out_pool: |
| 1273 | intel_engine_pool_put(pool); | 1375 | intel_gt_buffer_pool_put(pool); |
| 1274 | return err; | 1376 | return err; |
| 1275 | } | 1377 | } |
| 1276 | 1378 | ||
| 1379 | static bool reloc_can_use_engine(const struct intel_engine_cs *engine) | ||
| 1380 | { | ||
| 1381 | return engine->class != VIDEO_DECODE_CLASS || !IS_GEN(engine->i915, 6); | ||
| 1382 | } | ||
| 1383 | |||
| 1277 | static u32 *reloc_gpu(struct i915_execbuffer *eb, | 1384 | static u32 *reloc_gpu(struct i915_execbuffer *eb, |
| 1278 | struct i915_vma *vma, | 1385 | struct i915_vma *vma, |
| 1279 | unsigned int len) | 1386 | unsigned int len) |
| 1280 | { | 1387 | { |
| 1281 | struct reloc_cache *cache = &eb->reloc_cache; | 1388 | struct reloc_cache *cache = &eb->reloc_cache; |
| 1282 | u32 *cmd; | 1389 | u32 *cmd; |
| 1283 | 1390 | int err; | |
| 1284 | if (cache->rq_size > PAGE_SIZE/sizeof(u32) - (len + 1)) | ||
| 1285 | reloc_gpu_flush(cache); | ||
| 1286 | 1391 | ||
| 1287 | if (unlikely(!cache->rq)) { | 1392 | if (unlikely(!cache->rq)) { |
| 1288 | int err; | 1393 | struct intel_engine_cs *engine = eb->engine; |
| 1289 | 1394 | ||
| 1290 | if (!intel_engine_can_store_dword(eb->engine)) | 1395 | if (!reloc_can_use_engine(engine)) { |
| 1291 | return ERR_PTR(-ENODEV); | 1396 | engine = engine->gt->engine_class[COPY_ENGINE_CLASS][0]; |
| 1397 | if (!engine) | ||
| 1398 | return ERR_PTR(-ENODEV); | ||
| 1399 | } | ||
| 1292 | 1400 | ||
| 1293 | err = __reloc_gpu_alloc(eb, vma, len); | 1401 | err = __reloc_gpu_alloc(eb, engine, len); |
| 1294 | if (unlikely(err)) | 1402 | if (unlikely(err)) |
| 1295 | return ERR_PTR(err); | 1403 | return ERR_PTR(err); |
| 1296 | } | 1404 | } |
| 1297 | 1405 | ||
| 1406 | if (vma != cache->target) { | ||
| 1407 | err = reloc_move_to_gpu(cache->rq, vma); | ||
| 1408 | if (unlikely(err)) { | ||
| 1409 | i915_request_set_error_once(cache->rq, err); | ||
| 1410 | return ERR_PTR(err); | ||
| 1411 | } | ||
| 1412 | |||
| 1413 | cache->target = vma; | ||
| 1414 | } | ||
| 1415 | |||
| 1416 | if (unlikely(cache->rq_size + len > | ||
| 1417 | PAGE_SIZE / sizeof(u32) - RELOC_TAIL)) { | ||
| 1418 | err = reloc_gpu_chain(cache); | ||
| 1419 | if (unlikely(err)) { | ||
| 1420 | i915_request_set_error_once(cache->rq, err); | ||
| 1421 | return ERR_PTR(err); | ||
| 1422 | } | ||
| 1423 | } | ||
| 1424 | |||
| 1425 | GEM_BUG_ON(cache->rq_size + len >= PAGE_SIZE / sizeof(u32)); | ||
| 1298 | cmd = cache->rq_cmd + cache->rq_size; | 1426 | cmd = cache->rq_cmd + cache->rq_size; |
| 1299 | cache->rq_size += len; | 1427 | cache->rq_size += len; |
| 1300 | 1428 | ||
| @@ -1312,91 +1440,138 @@ static inline bool use_reloc_gpu(struct i915_vma *vma) · 11 unmodified lines | |||
| 1312 | return !dma_resv_test_signaled_rcu(vma->resv, true); | 1440 | return !dma_resv_test_signaled_rcu(vma->resv, true); |
| 1313 | } | 1441 | } |
| 1314 | 1442 | ||
| 1315 | static u64 | 1443 | static unsigned long vma_phys_addr(struct i915_vma *vma, u32 offset) |
| 1316 | relocate_entry(struct i915_vma *vma, | ||
| 1317 | const struct drm_i915_gem_relocation_entry *reloc, | ||
| 1318 | struct i915_execbuffer *eb, | ||
| 1319 | const struct i915_vma *target) | ||
| 1320 | { | 1444 | { |
| 1321 | u64 offset = reloc->offset; | 1445 | struct page *page; |
| 1322 | u64 target_offset = relocation_target(reloc, target); | 1446 | unsigned long addr; |
| 1323 | bool wide = eb->reloc_cache.use_64bit_reloc; | ||
| 1324 | void *vaddr; | ||
| 1325 | 1447 | ||
| 1326 | if (!eb->reloc_cache.vaddr && use_reloc_gpu(vma)) { | 1448 | GEM_BUG_ON(vma->pages != vma->obj->mm.pages); |
| 1327 | const unsigned int gen = eb->reloc_cache.gen; | ||
| 1328 | unsigned int len; | ||
| 1329 | u32 *batch; | ||
| 1330 | u64 addr; | ||
| 1331 | 1449 | ||
| 1332 | if (wide) | 1450 | page = i915_gem_object_get_page(vma->obj, offset >> PAGE_SHIFT); |
| 1333 | len = offset & 7 ? 8 : 5; | 1451 | addr = PFN_PHYS(page_to_pfn(page)); |
| 1334 | else if (gen >= 4) | 1452 | GEM_BUG_ON(overflows_type(addr, u32)); /* expected dma32 */ |
| 1335 | len = 4; | ||
| 1336 | else | ||
| 1337 | len = 3; | ||
| 1338 | 1453 | ||
| 1339 | batch = reloc_gpu(eb, vma, len); | 1454 | return addr + offset_in_page(offset); |
| 1340 | if (IS_ERR(batch)) | 1455 | } |
| 1341 | goto repeat; | 1456 | |
| 1457 | static bool __reloc_entry_gpu(struct i915_execbuffer *eb, | ||
| 1458 | struct i915_vma *vma, | ||
| 1459 | u64 offset, | ||
| 1460 | u64 target_addr) | ||
| 1461 | { | ||
| 1462 | const unsigned int gen = eb->reloc_cache.gen; | ||
| 1463 | unsigned int len; | ||
| 1464 | u32 *batch; | ||
| 1465 | u64 addr; | ||
| 1466 | |||
| 1467 | if (gen >= 8) | ||
| 1468 | len = offset & 7 ? 8 : 5; | ||
| 1469 | else if (gen >= 4) | ||
| 1470 | len = 4; | ||
| 1471 | else | ||
| 1472 | len = 3; | ||
| 1473 | |||
| 1474 | batch = reloc_gpu(eb, vma, len); | ||
| 1475 | if (IS_ERR(batch)) | ||
| 1476 | return false; | ||
| 1477 | |||
| 1478 | addr = gen8_canonical_addr(vma->node.start + offset); | ||
| 1479 | if (gen >= 8) { | ||
| 1480 | if (offset & 7) { | ||
| 1481 | *batch++ = MI_STORE_DWORD_IMM_GEN4; | ||
| 1482 | *batch++ = lower_32_bits(addr); | ||
| 1483 | *batch++ = upper_32_bits(addr); | ||
| 1484 | *batch++ = lower_32_bits(target_addr); | ||
| 1485 | |||
| 1486 | addr = gen8_canonical_addr(addr + 4); | ||
| 1342 | 1487 | ||
| 1343 | addr = gen8_canonical_addr(vma->node.start + offset); | ||
| 1344 | if (wide) { | ||
| 1345 | if (offset & 7) { | ||
| 1346 | *batch++ = MI_STORE_DWORD_IMM_GEN4; | ||
| 1347 | *batch++ = lower_32_bits(addr); | ||
| 1348 | *batch++ = upper_32_bits(addr); | ||
| 1349 | *batch++ = lower_32_bits(target_offset); | ||
| 1350 | |||
| 1351 | addr = gen8_canonical_addr(addr + 4); | ||
| 1352 | |||
| 1353 | *batch++ = MI_STORE_DWORD_IMM_GEN4; | ||
| 1354 | *batch++ = lower_32_bits(addr); | ||
| 1355 | *batch++ = upper_32_bits(addr); | ||
| 1356 | *batch++ = upper_32_bits(target_offset); | ||
| 1357 | } else { | ||
| 1358 | *batch++ = (MI_STORE_DWORD_IMM_GEN4 | (1 << 21)) + 1; | ||
| 1359 | *batch++ = lower_32_bits(addr); | ||
| 1360 | *batch++ = upper_32_bits(addr); | ||
| 1361 | *batch++ = lower_32_bits(target_offset); | ||
| 1362 | *batch++ = upper_32_bits(target_offset); | ||
| 1363 | } | ||
| 1364 | } else if (gen >= 6) { | ||
| 1365 | *batch++ = MI_STORE_DWORD_IMM_GEN4; | 1488 | *batch++ = MI_STORE_DWORD_IMM_GEN4; |
| 1366 | *batch++ = 0; | 1489 | *batch++ = lower_32_bits(addr); |
| 1367 | *batch++ = addr; | 1490 | *batch++ = upper_32_bits(addr); |
| 1368 | *batch++ = target_offset; | 1491 | *batch++ = upper_32_bits(target_addr); |
| 1369 | } else if (gen >= 4) { | ||
| 1370 | *batch++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT; | ||
| 1371 | *batch++ = 0; | ||
| 1372 | *batch++ = addr; | ||
| 1373 | *batch++ = target_offset; | ||
| 1374 | } else { | 1492 | } else { |
| 1375 | *batch++ = MI_STORE_DWORD_IMM | MI_MEM_VIRTUAL; | 1493 | *batch++ = (MI_STORE_DWORD_IMM_GEN4 | (1 << 21)) + 1; |
| 1376 | *batch++ = addr; | 1494 | *batch++ = lower_32_bits(addr); |
| 1377 | *batch++ = target_offset; | 1495 | *batch++ = upper_32_bits(addr); |
| 1496 | *batch++ = lower_32_bits(target_addr); | ||
| 1497 | *batch++ = upper_32_bits(target_addr); | ||
| 1378 | } | 1498 | } |
| 1379 | 1499 | } else if (gen >= 6) { | |
| 1380 | goto out; | 1500 | *batch++ = MI_STORE_DWORD_IMM_GEN4; |
| 1501 | *batch++ = 0; | ||
| 1502 | *batch++ = addr; | ||
| 1503 | *batch++ = target_addr; | ||
| 1504 | } else if (IS_I965G(eb->i915)) { | ||
| 1505 | *batch++ = MI_STORE_DWORD_IMM_GEN4; | ||
| 1506 | *batch++ = 0; | ||
| 1507 | *batch++ = vma_phys_addr(vma, offset); | ||
| 1508 | *batch++ = target_addr; | ||
| 1509 | } else if (gen >= 4) { | ||
| 1510 | *batch++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT; | ||
| 1511 | *batch++ = 0; | ||
| 1512 | *batch++ = addr; | ||
| 1513 | *batch++ = target_addr; | ||
| 1514 | } else if (gen >= 3 && | ||
| 1515 | !(IS_I915G(eb->i915) || IS_I915GM(eb->i915))) { | ||
| 1516 | *batch++ = MI_STORE_DWORD_IMM | MI_MEM_VIRTUAL; | ||
| 1517 | *batch++ = addr; | ||
| 1518 | *batch++ = target_addr; | ||
| 1519 | } else { | ||
| 1520 | *batch++ = MI_STORE_DWORD_IMM; | ||
| 1521 | *batch++ = vma_phys_addr(vma, offset); | ||
| 1522 | *batch++ = target_addr; | ||
| 1381 | } | 1523 | } |
| 1382 | 1524 | ||
| 1525 | return true; | ||
| 1526 | } | ||
| 1527 | |||
| 1528 | static bool reloc_entry_gpu(struct i915_execbuffer *eb, | ||
| 1529 | struct i915_vma *vma, | ||
| 1530 | u64 offset, | ||
| 1531 | u64 target_addr) | ||
| 1532 | { | ||
| 1533 | if (eb->reloc_cache.vaddr) | ||
| 1534 | return false; | ||
| 1535 | |||
| 1536 | if (!use_reloc_gpu(vma)) | ||
| 1537 | return false; | ||
| 1538 | |||
| 1539 | return __reloc_entry_gpu(eb, vma, offset, target_addr); | ||
| 1540 | } | ||
| 1541 | |||
| 1542 | static u64 | ||
| 1543 | relocate_entry(struct i915_vma *vma, | ||
| 1544 | const struct drm_i915_gem_relocation_entry *reloc, | ||
| 1545 | struct i915_execbuffer *eb, | ||
| 1546 | const struct i915_vma *target) | ||
| 1547 | { | ||
| 1548 | u64 target_addr = relocation_target(reloc, target); | ||
| 1549 | u64 offset = reloc->offset; | ||
| 1550 | |||
| 1551 | if (!reloc_entry_gpu(eb, vma, offset, target_addr)) { | ||
| 1552 | bool wide = eb->reloc_cache.use_64bit_reloc; | ||
| 1553 | void *vaddr; | ||
| 1554 | |||
| 1383 | repeat: | 1555 | repeat: |
| 1384 | vaddr = reloc_vaddr(vma->obj, &eb->reloc_cache, offset >> PAGE_SHIFT); | 1556 | vaddr = reloc_vaddr(vma->obj, |
| 1385 | if (IS_ERR(vaddr)) | 1557 | &eb->reloc_cache, |
| 1386 | return PTR_ERR(vaddr); | 1558 | offset >> PAGE_SHIFT); |
| 1559 | if (IS_ERR(vaddr)) | ||
| 1560 | return PTR_ERR(vaddr); | ||
| 1387 | 1561 | ||
| 1388 | clflush_write32(vaddr + offset_in_page(offset), | 1562 | GEM_BUG_ON(!IS_ALIGNED(offset, sizeof(u32))); |
| 1389 | lower_32_bits(target_offset), | 1563 | clflush_write32(vaddr + offset_in_page(offset), |
| 1390 | eb->reloc_cache.vaddr); | 1564 | lower_32_bits(target_addr), |
| 1565 | eb->reloc_cache.vaddr); | ||
| 1391 | 1566 | ||
| 1392 | if (wide) { | 1567 | if (wide) { |
| 1393 | offset += sizeof(u32); | 1568 | offset += sizeof(u32); |
| 1394 | target_offset >>= 32; | 1569 | target_addr >>= 32; |
| 1395 | wide = false; | 1570 | wide = false; |
| 1396 | goto repeat; | 1571 | goto repeat; |
| 1572 | } | ||
| 1397 | } | 1573 | } |
| 1398 | 1574 | ||
| 1399 | out: | ||
| 1400 | return target->node.start | UPDATE; | 1575 | return target->node.start | UPDATE; |
| 1401 | } | 1576 | } |
| 1402 | 1577 | ||
| @@ -1596,15 +1771,20 @@ static int eb_relocate(struct i915_execbuffer *eb) · 193 unmodified lines | |||
| 1596 | /* The objects are in their final locations, apply the relocations. */ | 1771 | /* The objects are in their final locations, apply the relocations. */ |
| 1597 | if (eb->args->flags & __EXEC_HAS_RELOC) { | 1772 | if (eb->args->flags & __EXEC_HAS_RELOC) { |
| 1598 | struct eb_vma *ev; | 1773 | struct eb_vma *ev; |
| 1774 | int flush; | ||
| 1599 | 1775 | ||
| 1600 | list_for_each_entry(ev, &eb->relocs, reloc_link) { | 1776 | list_for_each_entry(ev, &eb->relocs, reloc_link) { |
| 1601 | err = eb_relocate_vma(eb, ev); | 1777 | err = eb_relocate_vma(eb, ev); |
| 1602 | if (err) | 1778 | if (err) |
| 1603 | return err; | 1779 | break; |
| 1604 | } | 1780 | } |
| 1781 | |||
| 1782 | flush = reloc_gpu_flush(&eb->reloc_cache); | ||
| 1783 | if (!err) | ||
| 1784 | err = flush; | ||
| 1605 | } | 1785 | } |
| 1606 | 1786 | ||
| 1607 | return 0; | 1787 | return err; |
| 1608 | } | 1788 | } |
| 1609 | 1789 | ||
| 1610 | static int eb_move_to_gpu(struct i915_execbuffer *eb) | 1790 | static int eb_move_to_gpu(struct i915_execbuffer *eb) |
| @@ -1887,7 +2067,7 @@ err_free: · 276 unmodified lines | |||
| 1887 | static int eb_parse(struct i915_execbuffer *eb) | 2067 | static int eb_parse(struct i915_execbuffer *eb) |
| 1888 | { | 2068 | { |
| 1889 | struct drm_i915_private *i915 = eb->i915; | 2069 | struct drm_i915_private *i915 = eb->i915; |
| 1890 | struct intel_engine_pool_node *pool; | 2070 | struct intel_gt_buffer_pool_node *pool; |
| 1891 | struct i915_vma *shadow, *trampoline; | 2071 | struct i915_vma *shadow, *trampoline; |
| 1892 | unsigned int len; | 2072 | unsigned int len; |
| 1893 | int err; | 2073 | int err; |
| @@ -1910,7 +2090,7 @@ static int eb_parse(struct i915_execbuffer *eb) · 16 unmodified lines | |||
| 1910 | len += I915_CMD_PARSER_TRAMPOLINE_SIZE; | 2090 | len += I915_CMD_PARSER_TRAMPOLINE_SIZE; |
| 1911 | } | 2091 | } |
| 1912 | 2092 | ||
| 1913 | pool = intel_engine_get_pool(eb->engine, len); | 2093 | pool = intel_gt_get_buffer_pool(eb->engine->gt, len); |
| 1914 | if (IS_ERR(pool)) | 2094 | if (IS_ERR(pool)) |
| 1915 | return PTR_ERR(pool); | 2095 | return PTR_ERR(pool); |
| 1916 | 2096 | ||
| @@ -1958,7 +2138,7 @@ err_trampoline: · 41 unmodified lines | |||
| 1958 | err_shadow: | 2138 | err_shadow: |
| 1959 | i915_vma_unpin(shadow); | 2139 | i915_vma_unpin(shadow); |
| 1960 | err: | 2140 | err: |
| 1961 | intel_engine_pool_put(pool); | 2141 | intel_gt_buffer_pool_put(pool); |
| 1962 | return err; | 2142 | return err; |
| 1963 | } | 2143 | } |
| 1964 | 2144 | ||
| @@ -2402,30 +2582,6 @@ static void eb_request_add(struct i915_execbuffer *eb) · 437 unmodified lines | |||
| 2402 | /* Check that the context wasn't destroyed before submission */ | 2582 | /* Check that the context wasn't destroyed before submission */ |
| 2403 | if (likely(!intel_context_is_closed(eb->context))) { | 2583 | if (likely(!intel_context_is_closed(eb->context))) { |
| 2404 | attr = eb->gem_context->sched; | 2584 | attr = eb->gem_context->sched; |
| 2405 | |||
| 2406 | /* | ||
| 2407 | * Boost actual workloads past semaphores! | ||
| 2408 | * | ||
| 2409 | * With semaphores we spin on one engine waiting for another, | ||
| 2410 | * simply to reduce the latency of starting our work when | ||
| 2411 | * the signaler completes. However, if there is any other | ||
| 2412 | * work that we could be doing on this engine instead, that | ||
| 2413 | * is better utilisation and will reduce the overall duration | ||
| 2414 | * of the current work. To avoid PI boosting a semaphore | ||
| 2415 | * far in the distance past over useful work, we keep a history | ||
| 2416 | * of any semaphore use along our dependency chain. | ||
| 2417 | */ | ||
| 2418 | if (!(rq->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN)) | ||
| 2419 | attr.priority |= I915_PRIORITY_NOSEMAPHORE; | ||
| 2420 | |||
| 2421 | /* | ||
| 2422 | * Boost priorities to new clients (new request flows). | ||
| 2423 | * | ||
| 2424 | * Allow interactive/synchronous clients to jump ahead of | ||
| 2425 | * the bulk clients. (FQ_CODEL) | ||
| 2426 | */ | ||
| 2427 | if (list_empty(&rq->sched.signalers_list)) | ||
| 2428 | attr.priority |= I915_PRIORITY_WAIT; | ||
| 2429 | } else { | 2585 | } else { |
| 2430 | /* Serialise with context_close via the add_to_timeline */ | 2586 | /* Serialise with context_close via the add_to_timeline */ |
| 2431 | i915_request_set_error_once(rq, -ENOENT); | 2587 | i915_request_set_error_once(rq, -ENOENT); |
| @@ -2451,7 +2607,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, · 19 unmodified lines | |||
| 2451 | struct drm_i915_private *i915 = to_i915(dev); | 2607 | struct drm_i915_private *i915 = to_i915(dev); |
| 2452 | struct i915_execbuffer eb; | 2608 | struct i915_execbuffer eb; |
| 2453 | struct dma_fence *in_fence = NULL; | 2609 | struct dma_fence *in_fence = NULL; |
| 2454 | struct dma_fence *exec_fence = NULL; | ||
| 2455 | struct sync_file *out_fence = NULL; | 2610 | struct sync_file *out_fence = NULL; |
| 2456 | struct i915_vma *batch; | 2611 | struct i915_vma *batch; |
| 2457 | int out_fence_fd = -1; | 2612 | int out_fence_fd = -1; |
| @@ -2494,30 +2649,22 @@ i915_gem_do_execbuffer(struct drm_device *dev, · 36 unmodified lines | |||
| 2494 | if (args->flags & I915_EXEC_IS_PINNED) | 2649 | if (args->flags & I915_EXEC_IS_PINNED) |
| 2495 | eb.batch_flags |= I915_DISPATCH_PINNED; | 2650 | eb.batch_flags |= I915_DISPATCH_PINNED; |
| 2496 | 2651 | ||
| 2497 | if (args->flags & I915_EXEC_FENCE_IN) { | 2652 | #define IN_FENCES (I915_EXEC_FENCE_IN | I915_EXEC_FENCE_SUBMIT) |
| 2653 | if (args->flags & IN_FENCES) { | ||
| 2654 | if ((args->flags & IN_FENCES) == IN_FENCES) | ||
| 2655 | return -EINVAL; | ||
| 2656 | |||
| 2498 | in_fence = sync_file_get_fence(lower_32_bits(args->rsvd2)); | 2657 | in_fence = sync_file_get_fence(lower_32_bits(args->rsvd2)); |
| 2499 | if (!in_fence) | 2658 | if (!in_fence) |
| 2500 | return -EINVAL; | 2659 | return -EINVAL; |
| 2501 | } | 2660 | } |
| 2502 | 2661 | #undef IN_FENCES | |
| 2503 | if (args->flags & I915_EXEC_FENCE_SUBMIT) { | ||
| 2504 | if (in_fence) { | ||
| 2505 | err = -EINVAL; | ||
| 2506 | goto err_in_fence; | ||
| 2507 | } | ||
| 2508 | |||
| 2509 | exec_fence = sync_file_get_fence(lower_32_bits(args->rsvd2)); | ||
| 2510 | if (!exec_fence) { | ||
| 2511 | err = -EINVAL; | ||
| 2512 | goto err_in_fence; | ||
| 2513 | } | ||
| 2514 | } | ||
| 2515 | 2662 | ||
| 2516 | if (args->flags & I915_EXEC_FENCE_OUT) { | 2663 | if (args->flags & I915_EXEC_FENCE_OUT) { |
| 2517 | out_fence_fd = get_unused_fd_flags(O_CLOEXEC); | 2664 | out_fence_fd = get_unused_fd_flags(O_CLOEXEC); |
| 2518 | if (out_fence_fd < 0) { | 2665 | if (out_fence_fd < 0) { |
| 2519 | err = out_fence_fd; | 2666 | err = out_fence_fd; |
| 2520 | goto err_exec_fence; | 2667 | goto err_in_fence; |
| 2521 | } | 2668 | } |
| 2522 | } | 2669 | } |
| 2523 | 2670 | ||
| @@ -2608,14 +2755,13 @@ i915_gem_do_execbuffer(struct drm_device *dev, · 84 unmodified lines | |||
| 2608 | } | 2755 | } |
| 2609 | 2756 | ||
| 2610 | if (in_fence) { | 2757 | if (in_fence) { |
| 2611 | err = i915_request_await_dma_fence(eb.request, in_fence); | 2758 | if (args->flags & I915_EXEC_FENCE_SUBMIT) |
| 2612 | if (err < 0) | 2759 | err = i915_request_await_execution(eb.request, |
| 2613 | goto err_request; | 2760 | in_fence, |
| 2614 | } | 2761 | eb.engine->bond_execute); |
| 2615 | 2762 | else | |
| 2616 | if (exec_fence) { | 2763 | err = i915_request_await_dma_fence(eb.request, |
| 2617 | err = i915_request_await_execution(eb.request, exec_fence, | 2764 | in_fence); |
| 2618 | eb.engine->bond_execute); | ||
| 2619 | if (err < 0) | 2765 | if (err < 0) |
| 2620 | goto err_request; | 2766 | goto err_request; |
| 2621 | } | 2767 | } |
| @@ -2643,7 +2789,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, · 21 unmodified lines | |||
| 2643 | */ | 2789 | */ |
| 2644 | eb.request->batch = batch; | 2790 | eb.request->batch = batch; |
| 2645 | if (batch->private) | 2791 | if (batch->private) |
| 2646 | intel_engine_pool_mark_active(batch->private, eb.request); | 2792 | intel_gt_buffer_pool_mark_active(batch->private, eb.request); |
| 2647 | 2793 | ||
| 2648 | trace_i915_request_queue(eb.request, eb.batch_flags); | 2794 | trace_i915_request_queue(eb.request, eb.batch_flags); |
| 2649 | err = eb_submit(&eb, batch); | 2795 | err = eb_submit(&eb, batch); |
| @@ -2672,7 +2818,7 @@ err_batch_unpin: · 22 unmodified lines | |||
| 2672 | i915_vma_unpin(batch); | 2818 | i915_vma_unpin(batch); |
| 2673 | err_parse: | 2819 | err_parse: |
| 2674 | if (batch->private) | 2820 | if (batch->private) |
| 2675 | intel_engine_pool_put(batch->private); | 2821 | intel_gt_buffer_pool_put(batch->private); |
| 2676 | err_vma: | 2822 | err_vma: |
| 2677 | if (eb.trampoline) | 2823 | if (eb.trampoline) |
| 2678 | i915_vma_unpin(eb.trampoline); | 2824 | i915_vma_unpin(eb.trampoline); |
| @@ -2684,8 +2830,6 @@ err_destroy: · 5 unmodified lines | |||
| 2684 | err_out_fence: | 2830 | err_out_fence: |
| 2685 | if (out_fence_fd != -1) | 2831 | if (out_fence_fd != -1) |
| 2686 | put_unused_fd(out_fence_fd); | 2832 | put_unused_fd(out_fence_fd); |
| 2687 | err_exec_fence: | ||
| 2688 | dma_fence_put(exec_fence); | ||
| 2689 | err_in_fence: | 2833 | err_in_fence: |
| 2690 | dma_fence_put(in_fence); | 2834 | dma_fence_put(in_fence); |
| 2691 | return err; | 2835 | return err; |
| @@ -2894,3 +3038,7 @@ end:; · 202 unmodified lines | |||
| 2894 | kvfree(exec2_list); | 3038 | kvfree(exec2_list); |
| 2895 | return err; | 3039 | return err; |
| 2896 | } | 3040 | } |
| 3041 | |||
| 3042 | #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST) | ||
| 3043 | #include "selftests/i915_gem_execbuffer.c" | ||
| 3044 | #endif | ||
| @@ -72,8 +72,8 @@ i915_gem_object_lock_fence(struct drm_i915_gem_object *obj) · 71 unmodified lines | |||
| 72 | 0, 0); | 72 | 0, 0); |
| 73 | 73 | ||
| 74 | if (i915_sw_fence_await_reservation(&stub->chain, | 74 | if (i915_sw_fence_await_reservation(&stub->chain, |
| 75 | obj->base.resv, NULL, | 75 | obj->base.resv, NULL, true, |
| 76 | true, I915_FENCE_TIMEOUT, | 76 | i915_fence_timeout(to_i915(obj->base.dev)), |
| 77 | I915_FENCE_GFP) < 0) | 77 | I915_FENCE_GFP) < 0) |
| 78 | goto err; | 78 | goto err; |
| 79 | 79 | ||
| @@ -396,6 +396,38 @@ err: · 395 unmodified lines | |||
| 396 | return i915_error_to_vmf_fault(ret); | 396 | return i915_error_to_vmf_fault(ret); |
| 397 | } | 397 | } |
| 398 | 398 | ||
| 399 | static int | ||
| 400 | vm_access(struct vm_area_struct *area, unsigned long addr, | ||
| 401 | void *buf, int len, int write) | ||
| 402 | { | ||
| 403 | struct i915_mmap_offset *mmo = area->vm_private_data; | ||
| 404 | struct drm_i915_gem_object *obj = mmo->obj; | ||
| 405 | void *vaddr; | ||
| 406 | |||
| 407 | if (i915_gem_object_is_readonly(obj) && write) | ||
| 408 | return -EACCES; | ||
| 409 | |||
| 410 | addr -= area->vm_start; | ||
| 411 | if (addr >= obj->base.size) | ||
| 412 | return -EINVAL; | ||
| 413 | |||
| 414 | /* As this is primarily for debugging, let's focus on simplicity */ | ||
| 415 | vaddr = i915_gem_object_pin_map(obj, I915_MAP_FORCE_WC); | ||
| 416 | if (IS_ERR(vaddr)) | ||
| 417 | return PTR_ERR(vaddr); | ||
| 418 | |||
| 419 | if (write) { | ||
| 420 | memcpy(vaddr + addr, buf, len); | ||
| 421 | __i915_gem_object_flush_map(obj, addr, len); | ||
| 422 | } else { | ||
| 423 | memcpy(buf, vaddr + addr, len); | ||
| 424 | } | ||
| 425 | |||
| 426 | i915_gem_object_unpin_map(obj); | ||
| 427 | |||
| 428 | return len; | ||
| 429 | } | ||
| 430 | |||
| 399 | void __i915_gem_object_release_mmap_gtt(struct drm_i915_gem_object *obj) | 431 | void __i915_gem_object_release_mmap_gtt(struct drm_i915_gem_object *obj) |
| 400 | { | 432 | { |
| 401 | struct i915_vma *vma; | 433 | struct i915_vma *vma; |
| @@ -745,12 +777,14 @@ static void vm_close(struct vm_area_struct *vma) · 343 unmodified lines | |||
| 745 | 777 | ||
| 746 | static const struct vm_operations_struct vm_ops_gtt = { | 778 | static const struct vm_operations_struct vm_ops_gtt = { |
| 747 | .fault = vm_fault_gtt, | 779 | .fault = vm_fault_gtt, |
| 780 | .access = vm_access, | ||
| 748 | .open = vm_open, | 781 | .open = vm_open, |
| 749 | .close = vm_close, | 782 | .close = vm_close, |
| 750 | }; | 783 | }; |
| 751 | 784 | ||
| 752 | static const struct vm_operations_struct vm_ops_cpu = { | 785 | static const struct vm_operations_struct vm_ops_cpu = { |
| 753 | .fault = vm_fault_cpu, | 786 | .fault = vm_fault_cpu, |
| 787 | .access = vm_access, | ||
| 754 | .open = vm_open, | 788 | .open = vm_open, |
| 755 | .close = vm_close, | 789 | .close = vm_close, |
| 756 | }; | 790 | }; |
| @@ -162,9 +162,7 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915, · 161 unmodified lines | |||
| 162 | struct llist_node *freed) | 162 | struct llist_node *freed) |
| 163 | { | 163 | { |
| 164 | struct drm_i915_gem_object *obj, *on; | 164 | struct drm_i915_gem_object *obj, *on; |
| 165 | intel_wakeref_t wakeref; | ||
| 166 | 165 | ||
| 167 | wakeref = intel_runtime_pm_get(&i915->runtime_pm); | ||
| 168 | llist_for_each_entry_safe(obj, on, freed, freed) { | 166 | llist_for_each_entry_safe(obj, on, freed, freed) { |
| 169 | struct i915_mmap_offset *mmo, *mn; | 167 | struct i915_mmap_offset *mmo, *mn; |
| 170 | 168 | ||
| @@ -224,7 +222,6 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915, · 53 unmodified lines | |||
| 224 | call_rcu(&obj->rcu, __i915_gem_free_object_rcu); | 222 | call_rcu(&obj->rcu, __i915_gem_free_object_rcu); |
| 225 | cond_resched(); | 223 | cond_resched(); |
| 226 | } | 224 | } |
| 227 | intel_runtime_pm_put(&i915->runtime_pm, wakeref); | ||
| 228 | } | 225 | } |
| 229 | 226 | ||
| 230 | void i915_gem_flush_free_objects(struct drm_i915_private *i915) | 227 | void i915_gem_flush_free_objects(struct drm_i915_private *i915) |
| @@ -6,8 +6,8 @@ · 5 unmodified lines | |||
| 6 | #include "i915_drv.h" | 6 | #include "i915_drv.h" |
| 7 | #include "gt/intel_context.h" | 7 | #include "gt/intel_context.h" |
| 8 | #include "gt/intel_engine_pm.h" | 8 | #include "gt/intel_engine_pm.h" |
| 9 | #include "gt/intel_engine_pool.h" | ||
| 10 | #include "gt/intel_gt.h" | 9 | #include "gt/intel_gt.h" |
| 10 | #include "gt/intel_gt_buffer_pool.h" | ||
| 11 | #include "gt/intel_ring.h" | 11 | #include "gt/intel_ring.h" |
| 12 | #include "i915_gem_clflush.h" | 12 | #include "i915_gem_clflush.h" |
| 13 | #include "i915_gem_object_blt.h" | 13 | #include "i915_gem_object_blt.h" |
| @@ -18,7 +18,7 @@ struct i915_vma *intel_emit_vma_fill_blt(struct intel_context *ce, · 4 unmodified lines | |||
| 18 | { | 18 | { |
| 19 | struct drm_i915_private *i915 = ce->vm->i915; | 19 | struct drm_i915_private *i915 = ce->vm->i915; |
| 20 | const u32 block_size = SZ_8M; /* ~1ms at 8GiB/s preemption delay */ | 20 | const u32 block_size = SZ_8M; /* ~1ms at 8GiB/s preemption delay */ |
| 21 | struct intel_engine_pool_node *pool; | 21 | struct intel_gt_buffer_pool_node *pool; |
| 22 | struct i915_vma *batch; | 22 | struct i915_vma *batch; |
| 23 | u64 offset; | 23 | u64 offset; |
| 24 | u64 count; | 24 | u64 count; |
| @@ -33,7 +33,7 @@ struct i915_vma *intel_emit_vma_fill_blt(struct intel_context *ce, · 8 unmodified lines | |||
| 33 | count = div_u64(round_up(vma->size, block_size), block_size); | 33 | count = div_u64(round_up(vma->size, block_size), block_size); |
| 34 | size = (1 + 8 * count) * sizeof(u32); | 34 | size = (1 + 8 * count) * sizeof(u32); |
| 35 | size = round_up(size, PAGE_SIZE); | 35 | size = round_up(size, PAGE_SIZE); |
| 36 | pool = intel_engine_get_pool(ce->engine, size); | 36 | pool = intel_gt_get_buffer_pool(ce->engine->gt, size); |
| 37 | if (IS_ERR(pool)) { | 37 | if (IS_ERR(pool)) { |
| 38 | err = PTR_ERR(pool); | 38 | err = PTR_ERR(pool); |
| 39 | goto out_pm; | 39 | goto out_pm; |
| @@ -78,10 +78,12 @@ struct i915_vma *intel_emit_vma_fill_blt(struct intel_context *ce, · 38 unmodified lines | |||
| 78 | } while (rem); | 78 | } while (rem); |
| 79 | 79 | ||
| 80 | *cmd = MI_BATCH_BUFFER_END; | 80 | *cmd = MI_BATCH_BUFFER_END; |
| 81 | intel_gt_chipset_flush(ce->vm->gt); | ||
| 82 | 81 | ||
| 82 | i915_gem_object_flush_map(pool->obj); | ||
| 83 | i915_gem_object_unpin_map(pool->obj); | 83 | i915_gem_object_unpin_map(pool->obj); |
| 84 | 84 | ||
| 85 | intel_gt_chipset_flush(ce->vm->gt); | ||
| 86 | |||
| 85 | batch = i915_vma_instance(pool->obj, ce->vm, NULL); | 87 | batch = i915_vma_instance(pool->obj, ce->vm, NULL); |
| 86 | if (IS_ERR(batch)) { | 88 | if (IS_ERR(batch)) { |
| 87 | err = PTR_ERR(batch); | 89 | err = PTR_ERR(batch); |
| @@ -96,7 +98,7 @@ struct i915_vma *intel_emit_vma_fill_blt(struct intel_context *ce, · 8 unmodified lines | |||
| 96 | return batch; | 98 | return batch; |
| 97 | 99 | ||
| 98 | out_put: | 100 | out_put: |
| 99 | intel_engine_pool_put(pool); | 101 | intel_gt_buffer_pool_put(pool); |
| 100 | out_pm: | 102 | out_pm: |
| 101 | intel_engine_pm_put(ce->engine); | 103 | intel_engine_pm_put(ce->engine); |
| 102 | return ERR_PTR(err); | 104 | return ERR_PTR(err); |
| @@ -114,13 +116,13 @@ int intel_emit_vma_mark_active(struct i915_vma *vma, struct i915_request *rq) · 11 unmodified lines | |||
| 114 | if (unlikely(err)) | 116 | if (unlikely(err)) |
| 115 | return err; | 117 | return err; |
| 116 | 118 | ||
| 117 | return intel_engine_pool_mark_active(vma->private, rq); | 119 | return intel_gt_buffer_pool_mark_active(vma->private, rq); |
| 118 | } | 120 | } |
| 119 | 121 | ||
| 120 | void intel_emit_vma_release(struct intel_context *ce, struct i915_vma *vma) | 122 | void intel_emit_vma_release(struct intel_context *ce, struct i915_vma *vma) |
| 121 | { | 123 | { |
| 122 | i915_vma_unpin(vma); | 124 | i915_vma_unpin(vma); |
| 123 | intel_engine_pool_put(vma->private); | 125 | intel_gt_buffer_pool_put(vma->private); |
| 124 | intel_engine_pm_put(ce->engine); | 126 | intel_engine_pm_put(ce->engine); |
| 125 | } | 127 | } |
| 126 | 128 | ||
| @@ -213,7 +215,7 @@ struct i915_vma *intel_emit_vma_copy_blt(struct intel_context *ce, · 86 unmodified lines | |||
| 213 | { | 215 | { |
| 214 | struct drm_i915_private *i915 = ce->vm->i915; | 216 | struct drm_i915_private *i915 = ce->vm->i915; |
| 215 | const u32 block_size = SZ_8M; /* ~1ms at 8GiB/s preemption delay */ | 217 | const u32 block_size = SZ_8M; /* ~1ms at 8GiB/s preemption delay */ |
| 216 | struct intel_engine_pool_node *pool; | 218 | struct intel_gt_buffer_pool_node *pool; |
| 217 | struct i915_vma *batch; | 219 | struct i915_vma *batch; |
| 218 | u64 src_offset, dst_offset; | 220 | u64 src_offset, dst_offset; |
| 219 | u64 count, rem; | 221 | u64 count, rem; |
| @@ -228,7 +230,7 @@ struct i915_vma *intel_emit_vma_copy_blt(struct intel_context *ce, · 8 unmodified lines | |||
| 228 | count = div_u64(round_up(dst->size, block_size), block_size); | 230 | count = div_u64(round_up(dst->size, block_size), block_size); |
| 229 | size = (1 + 11 * count) * sizeof(u32); | 231 | size = (1 + 11 * count) * sizeof(u32); |
| 230 | size = round_up(size, PAGE_SIZE); | 232 | size = round_up(size, PAGE_SIZE); |
| 231 | pool = intel_engine_get_pool(ce->engine, size); | 233 | pool = intel_gt_get_buffer_pool(ce->engine->gt, size); |
| 232 | if (IS_ERR(pool)) { | 234 | if (IS_ERR(pool)) { |
| 233 | err = PTR_ERR(pool); | 235 | err = PTR_ERR(pool); |
| 234 | goto out_pm; | 236 | goto out_pm; |
| @@ -289,10 +291,12 @@ struct i915_vma *intel_emit_vma_copy_blt(struct intel_context *ce, · 54 unmodified lines | |||
| 289 | } while (rem); | 291 | } while (rem); |
| 290 | 292 | ||
| 291 | *cmd = MI_BATCH_BUFFER_END; | 293 | *cmd = MI_BATCH_BUFFER_END; |
| 292 | intel_gt_chipset_flush(ce->vm->gt); | ||
| 293 | 294 | ||
| 295 | i915_gem_object_flush_map(pool->obj); | ||
| 294 | i915_gem_object_unpin_map(pool->obj); | 296 | i915_gem_object_unpin_map(pool->obj); |
| 295 | 297 | ||
| 298 | intel_gt_chipset_flush(ce->vm->gt); | ||
| 299 | |||
| 296 | batch = i915_vma_instance(pool->obj, ce->vm, NULL); | 300 | batch = i915_vma_instance(pool->obj, ce->vm, NULL); |
| 297 | if (IS_ERR(batch)) { | 301 | if (IS_ERR(batch)) { |
| 298 | err = PTR_ERR(batch); | 302 | err = PTR_ERR(batch); |
| @@ -307,7 +311,7 @@ struct i915_vma *intel_emit_vma_copy_blt(struct intel_context *ce, · 8 unmodified lines | |||
| 307 | return batch; | 311 | return batch; |
| 308 | 312 | ||
| 309 | out_put: | 313 | out_put: |
| 310 | intel_engine_pool_put(pool); | 314 | intel_gt_buffer_pool_put(pool); |
| 311 | out_pm: | 315 | out_pm: |
| 312 | intel_engine_pm_put(ce->engine); | 316 | intel_engine_pm_put(ce->engine); |
| 313 | return ERR_PTR(err); | 317 | return ERR_PTR(err); |
| @@ -10,7 +10,6 @@ · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | #include "gt/intel_context.h" | 11 | #include "gt/intel_context.h" |
| 12 | #include "gt/intel_engine_pm.h" | 12 | #include "gt/intel_engine_pm.h" |
| 13 | #include "gt/intel_engine_pool.h" | ||
| 14 | #include "i915_vma.h" | 13 | #include "i915_vma.h" |
| 15 | 14 | ||
| 16 | struct drm_i915_gem_object; | 15 | struct drm_i915_gem_object; |
| @@ -391,6 +391,7 @@ void __i915_gem_object_flush_map(struct drm_i915_gem_object *obj, · 390 unmodified lines | |||
| 391 | GEM_BUG_ON(range_overflows_t(typeof(obj->base.size), | 391 | GEM_BUG_ON(range_overflows_t(typeof(obj->base.size), |
| 392 | offset, size, obj->base.size)); | 392 | offset, size, obj->base.size)); |
| 393 | 393 | ||
| 394 | wmb(); /* let all previous writes be visible to coherent partners */ | ||
| 394 | obj->mm.dirty = true; | 395 | obj->mm.dirty = true; |
| 395 | 396 | ||
| 396 | if (obj->cache_coherent & I915_BO_CACHE_COHERENT_FOR_WRITE) | 397 | if (obj->cache_coherent & I915_BO_CACHE_COHERENT_FOR_WRITE) |
| @@ -302,35 +302,6 @@ static void fill_scratch(struct tiled_blits *t, u32 *vaddr, u32 val) · 301 unmodified lines | |||
| 302 | i915_gem_object_flush_map(t->scratch.vma->obj); | 302 | i915_gem_object_flush_map(t->scratch.vma->obj); |
| 303 | } | 303 | } |
| 304 | 304 | ||
| 305 | static void hexdump(const void *buf, size_t len) | ||
| 306 | { | ||
| 307 | const size_t rowsize = 8 * sizeof(u32); | ||
| 308 | const void *prev = NULL; | ||
| 309 | bool skip = false; | ||
| 310 | size_t pos; | ||
| 311 | |||
| 312 | for (pos = 0; pos < len; pos += rowsize) { | ||
| 313 | char line[128]; | ||
| 314 | |||
| 315 | if (prev && !memcmp(prev, buf + pos, rowsize)) { | ||
| 316 | if (!skip) { | ||
| 317 | pr_info("*\n"); | ||
| 318 | skip = true; | ||
| 319 | } | ||
| 320 | continue; | ||
| 321 | } | ||
| 322 | |||
| 323 | WARN_ON_ONCE(hex_dump_to_buffer(buf + pos, len - pos, | ||
| 324 | rowsize, sizeof(u32), | ||
| 325 | line, sizeof(line), | ||
| 326 | false) >= sizeof(line)); | ||
| 327 | pr_info("[%04zx] %s\n", pos, line); | ||
| 328 | |||
| 329 | prev = buf + pos; | ||
| 330 | skip = false; | ||
| 331 | } | ||
| 332 | } | ||
| 333 | |||
| 334 | static u64 swizzle_bit(unsigned int bit, u64 offset) | 305 | static u64 swizzle_bit(unsigned int bit, u64 offset) |
| 335 | { | 306 | { |
| 336 | return (offset & BIT_ULL(bit)) >> (bit - 6); | 307 | return (offset & BIT_ULL(bit)) >> (bit - 6); |
| @@ -426,7 +397,7 @@ static int verify_buffer(const struct tiled_blits *t, · 89 unmodified lines | |||
| 426 | pr_err("Invalid %s tiling detected at (%d, %d), start_val %x\n", | 397 | pr_err("Invalid %s tiling detected at (%d, %d), start_val %x\n", |
| 427 | repr_tiling(buf->tiling), | 398 | repr_tiling(buf->tiling), |
| 428 | x, y, buf->start_val); | 399 | x, y, buf->start_val); |
| 429 | hexdump(vaddr, 4096); | 400 | igt_hexdump(vaddr, 4096); |
| 430 | } | 401 | } |
| 431 | 402 | ||
| 432 | i915_gem_object_unpin_map(buf->vma->obj); | 403 | i915_gem_object_unpin_map(buf->vma->obj); |
| @@ -158,6 +158,8 @@ static int wc_set(struct context *ctx, unsigned long offset, u32 v) · 157 unmodified lines | |||
| 158 | return PTR_ERR(map); | 158 | return PTR_ERR(map); |
| 159 | 159 | ||
| 160 | map[offset / sizeof(*map)] = v; | 160 | map[offset / sizeof(*map)] = v; |
| 161 | |||
| 162 | __i915_gem_object_flush_map(ctx->obj, offset, sizeof(*map)); | ||
| 161 | i915_gem_object_unpin_map(ctx->obj); | 163 | i915_gem_object_unpin_map(ctx->obj); |
| 162 | 164 | ||
| 163 | return 0; | 165 | return 0; |
| @@ -972,12 +972,6 @@ emit_rpcs_query(struct drm_i915_gem_object *obj, · 971 unmodified lines | |||
| 972 | goto err_batch; | 972 | goto err_batch; |
| 973 | } | 973 | } |
| 974 | 974 | ||
| 975 | err = rq->engine->emit_bb_start(rq, | ||
| 976 | batch->node.start, batch->node.size, | ||
| 977 | 0); | ||
| 978 | if (err) | ||
| 979 | goto err_request; | ||
| 980 | |||
| 981 | i915_vma_lock(batch); | 975 | i915_vma_lock(batch); |
| 982 | err = i915_request_await_object(rq, batch->obj, false); | 976 | err = i915_request_await_object(rq, batch->obj, false); |
| 983 | if (err == 0) | 977 | if (err == 0) |
| @@ -994,6 +988,18 @@ emit_rpcs_query(struct drm_i915_gem_object *obj, · 10 unmodified lines | |||
| 994 | if (err) | 988 | if (err) |
| 995 | goto skip_request; | 989 | goto skip_request; |
| 996 | 990 | ||
| 991 | if (rq->engine->emit_init_breadcrumb) { | ||
| 992 | err = rq->engine->emit_init_breadcrumb(rq); | ||
| 993 | if (err) | ||
| 994 | goto skip_request; | ||
| 995 | } | ||
| 996 | |||
| 997 | err = rq->engine->emit_bb_start(rq, | ||
| 998 | batch->node.start, batch->node.size, | ||
| 999 | 0); | ||
| 1000 | if (err) | ||
| 1001 | goto skip_request; | ||
| 1002 | |||
| 997 | i915_vma_unpin_and_release(&batch, 0); | 1003 | i915_vma_unpin_and_release(&batch, 0); |
| 998 | i915_vma_unpin(vma); | 1004 | i915_vma_unpin(vma); |
| 999 | 1005 | ||
| @@ -1005,7 +1011,6 @@ emit_rpcs_query(struct drm_i915_gem_object *obj, · 5 unmodified lines | |||
| 1005 | 1011 | ||
| 1006 | skip_request: | 1012 | skip_request: |
| 1007 | i915_request_set_error_once(rq, err); | 1013 | i915_request_set_error_once(rq, err); |
| 1008 | err_request: | ||
| 1009 | i915_request_add(rq); | 1014 | i915_request_add(rq); |
| 1010 | err_batch: | 1015 | err_batch: |
| 1011 | i915_vma_unpin_and_release(&batch, 0); | 1016 | i915_vma_unpin_and_release(&batch, 0); |
| @@ -1541,10 +1546,6 @@ static int write_to_scratch(struct i915_gem_context *ctx, · 529 unmodified lines | |||
| 1541 | goto err_unpin; | 1546 | goto err_unpin; |
| 1542 | } | 1547 | } |
| 1543 | 1548 | ||
| 1544 | err = engine->emit_bb_start(rq, vma->node.start, vma->node.size, 0); | ||
| 1545 | if (err) | ||
| 1546 | goto err_request; | ||
| 1547 | |||
| 1548 | i915_vma_lock(vma); | 1549 | i915_vma_lock(vma); |
| 1549 | err = i915_request_await_object(rq, vma->obj, false); | 1550 | err = i915_request_await_object(rq, vma->obj, false); |
| 1550 | if (err == 0) | 1551 | if (err == 0) |
| @@ -1553,6 +1554,16 @@ static int write_to_scratch(struct i915_gem_context *ctx, · 2 unmodified lines | |||
| 1553 | if (err) | 1554 | if (err) |
| 1554 | goto skip_request; | 1555 | goto skip_request; |
| 1555 | 1556 | ||
| 1557 | if (rq->engine->emit_init_breadcrumb) { | ||
| 1558 | err = rq->engine->emit_init_breadcrumb(rq); | ||
| 1559 | if (err) | ||
| 1560 | goto skip_request; | ||
| 1561 | } | ||
| 1562 | |||
| 1563 | err = engine->emit_bb_start(rq, vma->node.start, vma->node.size, 0); | ||
| 1564 | if (err) | ||
| 1565 | goto skip_request; | ||
| 1566 | |||
| 1556 | i915_vma_unpin(vma); | 1567 | i915_vma_unpin(vma); |
| 1557 | 1568 | ||
| 1558 | i915_request_add(rq); | 1569 | i915_request_add(rq); |
| @@ -1560,7 +1571,6 @@ static int write_to_scratch(struct i915_gem_context *ctx, · 1 unmodified line | |||
| 1560 | goto out_vm; | 1571 | goto out_vm; |
| 1561 | skip_request: | 1572 | skip_request: |
| 1562 | i915_request_set_error_once(rq, err); | 1573 | i915_request_set_error_once(rq, err); |
| 1563 | err_request: | ||
| 1564 | i915_request_add(rq); | 1574 | i915_request_add(rq); |
| 1565 | err_unpin: | 1575 | err_unpin: |
| 1566 | i915_vma_unpin(vma); | 1576 | i915_vma_unpin(vma); |
| @@ -1674,10 +1684,6 @@ static int read_from_scratch(struct i915_gem_context *ctx, · 107 unmodified lines | |||
| 1674 | goto err_unpin; | 1684 | goto err_unpin; |
| 1675 | } | 1685 | } |
| 1676 | 1686 | ||
| 1677 | err = engine->emit_bb_start(rq, vma->node.start, vma->node.size, flags); | ||
| 1678 | if (err) | ||
| 1679 | goto err_request; | ||
| 1680 | |||
| 1681 | i915_vma_lock(vma); | 1687 | i915_vma_lock(vma); |
| 1682 | err = i915_request_await_object(rq, vma->obj, true); | 1688 | err = i915_request_await_object(rq, vma->obj, true); |
| 1683 | if (err == 0) | 1689 | if (err == 0) |
| @@ -1686,6 +1692,16 @@ static int read_from_scratch(struct i915_gem_context *ctx, · 2 unmodified lines | |||
| 1686 | if (err) | 1692 | if (err) |
| 1687 | goto skip_request; | 1693 | goto skip_request; |
| 1688 | 1694 | ||
| 1695 | if (rq->engine->emit_init_breadcrumb) { | ||
| 1696 | err = rq->engine->emit_init_breadcrumb(rq); | ||
| 1697 | if (err) | ||
| 1698 | goto skip_request; | ||
| 1699 | } | ||
| 1700 | |||
| 1701 | err = engine->emit_bb_start(rq, vma->node.start, vma->node.size, flags); | ||
| 1702 | if (err) | ||
| 1703 | goto skip_request; | ||
| 1704 | |||
| 1689 | i915_vma_unpin(vma); | 1705 | i915_vma_unpin(vma); |
| 1690 | 1706 | ||
| 1691 | i915_request_add(rq); | 1707 | i915_request_add(rq); |
| @@ -1708,7 +1724,6 @@ static int read_from_scratch(struct i915_gem_context *ctx, · 16 unmodified lines | |||
| 1708 | goto out_vm; | 1724 | goto out_vm; |
| 1709 | skip_request: | 1725 | skip_request: |
| 1710 | i915_request_set_error_once(rq, err); | 1726 | i915_request_set_error_once(rq, err); |
| 1711 | err_request: | ||
| 1712 | i915_request_add(rq); | 1727 | i915_request_add(rq); |
| 1713 | err_unpin: | 1728 | err_unpin: |
| 1714 | i915_vma_unpin(vma); | 1729 | i915_vma_unpin(vma); |
| @@ -0,0 +1,171 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright © 2020 Intel Corporation | ||
| 4 | */ | ||
| 5 | |||
| 6 | #include "i915_selftest.h" | ||
| 7 | |||
| 8 | #include "gt/intel_engine_pm.h" | ||
| 9 | #include "selftests/igt_flush_test.h" | ||
| 10 | |||
| 11 | static u64 read_reloc(const u32 *map, int x, const u64 mask) | ||
| 12 | { | ||
| 13 | u64 reloc; | ||
| 14 | |||
| 15 | memcpy(&reloc, &map[x], sizeof(reloc)); | ||
| 16 | return reloc & mask; | ||
| 17 | } | ||
| 18 | |||
| 19 | static int __igt_gpu_reloc(struct i915_execbuffer *eb, | ||
| 20 | struct drm_i915_gem_object *obj) | ||
| 21 | { | ||
| 22 | const unsigned int offsets[] = { 8, 3, 0 }; | ||
| 23 | const u64 mask = | ||
| 24 | GENMASK_ULL(eb->reloc_cache.use_64bit_reloc ? 63 : 31, 0); | ||
| 25 | const u32 *map = page_mask_bits(obj->mm.mapping); | ||
| 26 | struct i915_request *rq; | ||
| 27 | struct i915_vma *vma; | ||
| 28 | int err; | ||
| 29 | int i; | ||
| 30 | |||
| 31 | vma = i915_vma_instance(obj, eb->context->vm, NULL); | ||
| 32 | if (IS_ERR(vma)) | ||
| 33 | return PTR_ERR(vma); | ||
| 34 | |||
| 35 | err = i915_vma_pin(vma, 0, 0, PIN_USER | PIN_HIGH); | ||
| 36 | if (err) | ||
| 37 | return err; | ||
| 38 | |||
| 39 | /* 8-Byte aligned */ | ||
| 40 | if (!__reloc_entry_gpu(eb, vma, | ||
| 41 | offsets[0] * sizeof(u32), | ||
| 42 | 0)) { | ||
| 43 | err = -EIO; | ||
| 44 | goto unpin_vma; | ||
| 45 | } | ||
| 46 | |||
| 47 | /* !8-Byte aligned */ | ||
| 48 | if (!__reloc_entry_gpu(eb, vma, | ||
| 49 | offsets[1] * sizeof(u32), | ||
| 50 | 1)) { | ||
| 51 | err = -EIO; | ||
| 52 | goto unpin_vma; | ||
| 53 | } | ||
| 54 | |||
| 55 | /* Skip to the end of the cmd page */ | ||
| 56 | i = PAGE_SIZE / sizeof(u32) - RELOC_TAIL - 1; | ||
| 57 | i -= eb->reloc_cache.rq_size; | ||
| 58 | memset32(eb->reloc_cache.rq_cmd + eb->reloc_cache.rq_size, | ||
| 59 | MI_NOOP, i); | ||
| 60 | eb->reloc_cache.rq_size += i; | ||
| 61 | |||
| 62 | /* Force batch chaining */ | ||
| 63 | if (!__reloc_entry_gpu(eb, vma, | ||
| 64 | offsets[2] * sizeof(u32), | ||
| 65 | 2)) { | ||
| 66 | err = -EIO; | ||
| 67 | goto unpin_vma; | ||
| 68 | } | ||
| 69 | |||
| 70 | GEM_BUG_ON(!eb->reloc_cache.rq); | ||
| 71 | rq = i915_request_get(eb->reloc_cache.rq); | ||
| 72 | err = reloc_gpu_flush(&eb->reloc_cache); | ||
| 73 | if (err) | ||
| 74 | goto put_rq; | ||
| 75 | GEM_BUG_ON(eb->reloc_cache.rq); | ||
| 76 | |||
| 77 | err = i915_gem_object_wait(obj, I915_WAIT_INTERRUPTIBLE, HZ / 2); | ||
| 78 | if (err) { | ||
| 79 | intel_gt_set_wedged(eb->engine->gt); | ||
| 80 | goto put_rq; | ||
| 81 | } | ||
| 82 | |||
| 83 | if (!i915_request_completed(rq)) { | ||
| 84 | pr_err("%s: did not wait for relocations!\n", eb->engine->name); | ||
| 85 | err = -EINVAL; | ||
| 86 | goto put_rq; | ||
| 87 | } | ||
| 88 | |||
| 89 | for (i = 0; i < ARRAY_SIZE(offsets); i++) { | ||
| 90 | u64 reloc = read_reloc(map, offsets[i], mask); | ||
| 91 | |||
| 92 | if (reloc != i) { | ||
| 93 | pr_err("%s[%d]: map[%d] %llx != %x\n", | ||
| 94 | eb->engine->name, i, offsets[i], reloc, i); | ||
| 95 | err = -EINVAL; | ||
| 96 | } | ||
| 97 | } | ||
| 98 | if (err) | ||
| 99 | igt_hexdump(map, 4096); | ||
| 100 | |||
| 101 | put_rq: | ||
| 102 | i915_request_put(rq); | ||
| 103 | unpin_vma: | ||
| 104 | i915_vma_unpin(vma); | ||
| 105 | return err; | ||
| 106 | } | ||
| 107 | |||
| 108 | static int igt_gpu_reloc(void *arg) | ||
| 109 | { | ||
| 110 | struct i915_execbuffer eb; | ||
| 111 | struct drm_i915_gem_object *scratch; | ||
| 112 | int err = 0; | ||
| 113 | u32 *map; | ||
| 114 | |||
| 115 | eb.i915 = arg; | ||
| 116 | |||
| 117 | scratch = i915_gem_object_create_internal(eb.i915, 4096); | ||
| 118 | if (IS_ERR(scratch)) | ||
| 119 | return PTR_ERR(scratch); | ||
| 120 | |||
| 121 | map = i915_gem_object_pin_map(scratch, I915_MAP_WC); | ||
| 122 | if (IS_ERR(map)) { | ||
| 123 | err = PTR_ERR(map); | ||
| 124 | goto err_scratch; | ||
| 125 | } | ||
| 126 | |||
| 127 | for_each_uabi_engine(eb.engine, eb.i915) { | ||
| 128 | reloc_cache_init(&eb.reloc_cache, eb.i915); | ||
| 129 | memset(map, POISON_INUSE, 4096); | ||
| 130 | |||
| 131 | intel_engine_pm_get(eb.engine); | ||
| 132 | eb.context = intel_context_create(eb.engine); | ||
| 133 | if (IS_ERR(eb.context)) { | ||
| 134 | err = PTR_ERR(eb.context); | ||
| 135 | goto err_pm; | ||
| 136 | } | ||
| 137 | |||
| 138 | err = intel_context_pin(eb.context); | ||
| 139 | if (err) | ||
| 140 | goto err_put; | ||
| 141 | |||
| 142 | err = __igt_gpu_reloc(&eb, scratch); | ||
| 143 | |||
| 144 | intel_context_unpin(eb.context); | ||
| 145 | err_put: | ||
| 146 | intel_context_put(eb.context); | ||
| 147 | err_pm: | ||
| 148 | intel_engine_pm_put(eb.engine); | ||
| 149 | if (err) | ||
| 150 | break; | ||
| 151 | } | ||
| 152 | |||
| 153 | if (igt_flush_test(eb.i915)) | ||
| 154 | err = -EIO; | ||
| 155 | |||
| 156 | err_scratch: | ||
| 157 | i915_gem_object_put(scratch); | ||
| 158 | return err; | ||
| 159 | } | ||
| 160 | |||
| 161 | int i915_gem_execbuffer_live_selftests(struct drm_i915_private *i915) | ||
| 162 | { | ||
| 163 | static const struct i915_subtest tests[] = { | ||
| 164 | SUBTEST(igt_gpu_reloc), | ||
| 165 | }; | ||
| 166 | |||
| 167 | if (intel_gt_is_wedged(&i915->gt)) | ||
| 168 | return 0; | ||
| 169 | |||
| 170 | return i915_live_subtests(tests, i915); | ||
| 171 | } | ||
| @@ -952,6 +952,129 @@ static int igt_mmap(void *arg) · 951 unmodified lines | |||
| 952 | return 0; | 952 | return 0; |
| 953 | } | 953 | } |
| 954 | 954 | ||
| 955 | static const char *repr_mmap_type(enum i915_mmap_type type) | ||
| 956 | { | ||
| 957 | switch (type) { | ||
| 958 | case I915_MMAP_TYPE_GTT: return "gtt"; | ||
| 959 | case I915_MMAP_TYPE_WB: return "wb"; | ||
| 960 | case I915_MMAP_TYPE_WC: return "wc"; | ||
| 961 | case I915_MMAP_TYPE_UC: return "uc"; | ||
| 962 | default: return "unknown"; | ||
| 963 | } | ||
| 964 | } | ||
| 965 | |||
| 966 | static bool can_access(const struct drm_i915_gem_object *obj) | ||
| 967 | { | ||
| 968 | unsigned int flags = | ||
| 969 | I915_GEM_OBJECT_HAS_STRUCT_PAGE | I915_GEM_OBJECT_HAS_IOMEM; | ||
| 970 | |||
| 971 | return i915_gem_object_type_has(obj, flags); | ||
| 972 | } | ||
| 973 | |||
| 974 | static int __igt_mmap_access(struct drm_i915_private *i915, | ||
| 975 | struct drm_i915_gem_object *obj, | ||
| 976 | enum i915_mmap_type type) | ||
| 977 | { | ||
| 978 | struct i915_mmap_offset *mmo; | ||
| 979 | unsigned long __user *ptr; | ||
| 980 | unsigned long A, B; | ||
| 981 | unsigned long x, y; | ||
| 982 | unsigned long addr; | ||
| 983 | int err; | ||
| 984 | |||
| 985 | memset(&A, 0xAA, sizeof(A)); | ||
| 986 | memset(&B, 0xBB, sizeof(B)); | ||
| 987 | |||
| 988 | if (!can_mmap(obj, type) || !can_access(obj)) | ||
| 989 | return 0; | ||
| 990 | |||
| 991 | mmo = mmap_offset_attach(obj, type, NULL); | ||
| 992 | if (IS_ERR(mmo)) | ||
| 993 | return PTR_ERR(mmo); | ||
| 994 | |||
| 995 | addr = igt_mmap_node(i915, &mmo->vma_node, 0, PROT_WRITE, MAP_SHARED); | ||
| 996 | if (IS_ERR_VALUE(addr)) | ||
| 997 | return addr; | ||
| 998 | ptr = (unsigned long __user *)addr; | ||
| 999 | |||
| 1000 | err = __put_user(A, ptr); | ||
| 1001 | if (err) { | ||
| 1002 | pr_err("%s(%s): failed to write into user mmap\n", | ||
| 1003 | obj->mm.region->name, repr_mmap_type(type)); | ||
| 1004 | goto out_unmap; | ||
| 1005 | } | ||
| 1006 | |||
| 1007 | intel_gt_flush_ggtt_writes(&i915->gt); | ||
| 1008 | |||
| 1009 | err = access_process_vm(current, addr, &x, sizeof(x), 0); | ||
| 1010 | if (err != sizeof(x)) { | ||
| 1011 | pr_err("%s(%s): access_process_vm() read failed\n", | ||
| 1012 | obj->mm.region->name, repr_mmap_type(type)); | ||
| 1013 | goto out_unmap; | ||
| 1014 | } | ||
| 1015 | |||
| 1016 | err = access_process_vm(current, addr, &B, sizeof(B), FOLL_WRITE); | ||
| 1017 | if (err != sizeof(B)) { | ||
| 1018 | pr_err("%s(%s): access_process_vm() write failed\n", | ||
| 1019 | obj->mm.region->name, repr_mmap_type(type)); | ||
| 1020 | goto out_unmap; | ||
| 1021 | } | ||
| 1022 | |||
| 1023 | intel_gt_flush_ggtt_writes(&i915->gt); | ||
| 1024 | |||
| 1025 | err = __get_user(y, ptr); | ||
| 1026 | if (err) { | ||
| 1027 | pr_err("%s(%s): failed to read from user mmap\n", | ||
| 1028 | obj->mm.region->name, repr_mmap_type(type)); | ||
| 1029 | goto out_unmap; | ||
| 1030 | } | ||
| 1031 | |||
| 1032 | if (x != A || y != B) { | ||
| 1033 | pr_err("%s(%s): failed to read/write values, found (%lx, %lx)\n", | ||
| 1034 | obj->mm.region->name, repr_mmap_type(type), | ||
| 1035 | x, y); | ||
| 1036 | err = -EINVAL; | ||
| 1037 | goto out_unmap; | ||
| 1038 | } | ||
| 1039 | |||
| 1040 | out_unmap: | ||
| 1041 | vm_munmap(addr, obj->base.size); | ||
| 1042 | return err; | ||
| 1043 | } | ||
| 1044 | |||
| 1045 | static int igt_mmap_access(void *arg) | ||
| 1046 | { | ||
| 1047 | struct drm_i915_private *i915 = arg; | ||
| 1048 | struct intel_memory_region *mr; | ||
| 1049 | enum intel_region_id id; | ||
| 1050 | |||
| 1051 | for_each_memory_region(mr, i915, id) { | ||
| 1052 | struct drm_i915_gem_object *obj; | ||
| 1053 | int err; | ||
| 1054 | |||
| 1055 | obj = i915_gem_object_create_region(mr, PAGE_SIZE, 0); | ||
| 1056 | if (obj == ERR_PTR(-ENODEV)) | ||
| 1057 | continue; | ||
| 1058 | |||
| 1059 | if (IS_ERR(obj)) | ||
| 1060 | return PTR_ERR(obj); | ||
| 1061 | |||
| 1062 | err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_GTT); | ||
| 1063 | if (err == 0) | ||
| 1064 | err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_WB); | ||
| 1065 | if (err == 0) | ||
| 1066 | err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_WC); | ||
| 1067 | if (err == 0) | ||
| 1068 | err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_UC); | ||
| 1069 | |||
| 1070 | i915_gem_object_put(obj); | ||
| 1071 | if (err) | ||
| 1072 | return err; | ||
| 1073 | } | ||
| 1074 | |||
| 1075 | return 0; | ||
| 1076 | } | ||
| 1077 | |||
| 955 | static int __igt_mmap_gpu(struct drm_i915_private *i915, | 1078 | static int __igt_mmap_gpu(struct drm_i915_private *i915, |
| 956 | struct drm_i915_gem_object *obj, | 1079 | struct drm_i915_gem_object *obj, |
| 957 | enum i915_mmap_type type) | 1080 | enum i915_mmap_type type) |
| @@ -1229,6 +1352,7 @@ int i915_gem_mman_live_selftests(struct drm_i915_private *i915) · 271 unmodified lines | |||
| 1229 | SUBTEST(igt_smoke_tiling), | 1352 | SUBTEST(igt_smoke_tiling), |
| 1230 | SUBTEST(igt_mmap_offset_exhaustion), | 1353 | SUBTEST(igt_mmap_offset_exhaustion), |
| 1231 | SUBTEST(igt_mmap), | 1354 | SUBTEST(igt_mmap), |
| 1355 | SUBTEST(igt_mmap_access), | ||
| 1232 | SUBTEST(igt_mmap_revoke), | 1356 | SUBTEST(igt_mmap_revoke), |
| 1233 | SUBTEST(igt_mmap_gpu), | 1357 | SUBTEST(igt_mmap_gpu), |
| 1234 | }; | 1358 | }; |
| @@ -83,6 +83,8 @@ igt_emit_store_dw(struct i915_vma *vma, · 82 unmodified lines | |||
| 83 | offset += PAGE_SIZE; | 83 | offset += PAGE_SIZE; |
| 84 | } | 84 | } |
| 85 | *cmd = MI_BATCH_BUFFER_END; | 85 | *cmd = MI_BATCH_BUFFER_END; |
| 86 | |||
| 87 | i915_gem_object_flush_map(obj); | ||
| 86 | i915_gem_object_unpin_map(obj); | 88 | i915_gem_object_unpin_map(obj); |
| 87 | 89 | ||
| 88 | intel_gt_chipset_flush(vma->vm->gt); | 90 | intel_gt_chipset_flush(vma->vm->gt); |
| @@ -126,16 +128,6 @@ int igt_gpu_fill_dw(struct intel_context *ce, · 37 unmodified lines | |||
| 126 | goto err_batch; | 128 | goto err_batch; |
| 127 | } | 129 | } |
| 128 | 130 | ||
| 129 | flags = 0; | ||
| 130 | if (INTEL_GEN(ce->vm->i915) <= 5) | ||
| 131 | flags |= I915_DISPATCH_SECURE; | ||
| 132 | |||
| 133 | err = rq->engine->emit_bb_start(rq, | ||
| 134 | batch->node.start, batch->node.size, | ||
| 135 | flags); | ||
| 136 | if (err) | ||
| 137 | goto err_request; | ||
| 138 | |||
| 139 | i915_vma_lock(batch); | 131 | i915_vma_lock(batch); |
| 140 | err = i915_request_await_object(rq, batch->obj, false); | 132 | err = i915_request_await_object(rq, batch->obj, false); |
| 141 | if (err == 0) | 133 | if (err == 0) |
| @@ -152,15 +144,17 @@ int igt_gpu_fill_dw(struct intel_context *ce, · 10 unmodified lines | |||
| 152 | if (err) | 144 | if (err) |
| 153 | goto skip_request; | 145 | goto skip_request; |
| 154 | 146 | ||
| 155 | i915_request_add(rq); | 147 | flags = 0; |
| 156 | 148 | if (INTEL_GEN(ce->vm->i915) <= 5) | |
| 157 | i915_vma_unpin_and_release(&batch, 0); | 149 | flags |= I915_DISPATCH_SECURE; |
| 158 | 150 | ||
| 159 | return 0; | 151 | err = rq->engine->emit_bb_start(rq, |
| 152 | batch->node.start, batch->node.size, | ||
| 153 | flags); | ||
| 160 | 154 | ||
| 161 | skip_request: | 155 | skip_request: |
| 162 | i915_request_set_error_once(rq, err); | 156 | if (err) |
| 163 | err_request: | 157 | i915_request_set_error_once(rq, err); |
| 164 | i915_request_add(rq); | 158 | i915_request_add(rq); |
| 165 | err_batch: | 159 | err_batch: |
| 166 | i915_vma_unpin_and_release(&batch, 0); | 160 | i915_vma_unpin_and_release(&batch, 0); |
| @@ -389,6 +389,16 @@ static int gen8_ppgtt_alloc(struct i915_address_space *vm, · 388 unmodified lines | |||
| 389 | return err; | 389 | return err; |
| 390 | } | 390 | } |
| 391 | 391 | ||
| 392 | static __always_inline void | ||
| 393 | write_pte(gen8_pte_t *pte, const gen8_pte_t val) | ||
| 394 | { | ||
| 395 | /* Magic delays? Or can we refine these to flush all in one pass? */ | ||
| 396 | *pte = val; | ||
| 397 | wmb(); /* cpu to cache */ | ||
| 398 | clflush(pte); /* cache to memory */ | ||
| 399 | wmb(); /* visible to all */ | ||
| 400 | } | ||
| 401 | |||
| 392 | static __always_inline u64 | 402 | static __always_inline u64 |
| 393 | gen8_ppgtt_insert_pte(struct i915_ppgtt *ppgtt, | 403 | gen8_ppgtt_insert_pte(struct i915_ppgtt *ppgtt, |
| 394 | struct i915_page_directory *pdp, | 404 | struct i915_page_directory *pdp, |
| @@ -405,7 +415,8 @@ gen8_ppgtt_insert_pte(struct i915_ppgtt *ppgtt, · 10 unmodified lines | |||
| 405 | vaddr = kmap_atomic_px(i915_pt_entry(pd, gen8_pd_index(idx, 1))); | 415 | vaddr = kmap_atomic_px(i915_pt_entry(pd, gen8_pd_index(idx, 1))); |
| 406 | do { | 416 | do { |
| 407 | GEM_BUG_ON(iter->sg->length < I915_GTT_PAGE_SIZE); | 417 | GEM_BUG_ON(iter->sg->length < I915_GTT_PAGE_SIZE); |
| 408 | vaddr[gen8_pd_index(idx, 0)] = pte_encode | iter->dma; | 418 | write_pte(&vaddr[gen8_pd_index(idx, 0)], |
| 419 | pte_encode | iter->dma); | ||
| 409 | 420 | ||
| 410 | iter->dma += I915_GTT_PAGE_SIZE; | 421 | iter->dma += I915_GTT_PAGE_SIZE; |
| 411 | if (iter->dma >= iter->max) { | 422 | if (iter->dma >= iter->max) { |
| @@ -487,7 +498,7 @@ static void gen8_ppgtt_insert_huge(struct i915_vma *vma, · 75 unmodified lines | |||
| 487 | 498 | ||
| 488 | do { | 499 | do { |
| 489 | GEM_BUG_ON(iter->sg->length < page_size); | 500 | GEM_BUG_ON(iter->sg->length < page_size); |
| 490 | vaddr[index++] = encode | iter->dma; | 501 | write_pte(&vaddr[index++], encode | iter->dma); |
| 491 | 502 | ||
| 492 | start += page_size; | 503 | start += page_size; |
| 493 | iter->dma += page_size; | 504 | iter->dma += page_size; |
| @@ -142,6 +142,18 @@ static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl) · 141 unmodified lines | |||
| 142 | intel_engine_add_retire(engine, tl); | 142 | intel_engine_add_retire(engine, tl); |
| 143 | } | 143 | } |
| 144 | 144 | ||
| 145 | static void __signal_request(struct i915_request *rq, struct list_head *signals) | ||
| 146 | { | ||
| 147 | GEM_BUG_ON(!test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags)); | ||
| 148 | clear_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags); | ||
| 149 | |||
| 150 | if (!__dma_fence_signal(&rq->fence)) | ||
| 151 | return; | ||
| 152 | |||
| 153 | i915_request_get(rq); | ||
| 154 | list_add_tail(&rq->signal_link, signals); | ||
| 155 | } | ||
| 156 | |||
| 145 | static void signal_irq_work(struct irq_work *work) | 157 | static void signal_irq_work(struct irq_work *work) |
| 146 | { | 158 | { |
| 147 | struct intel_breadcrumbs *b = container_of(work, typeof(*b), irq_work); | 159 | struct intel_breadcrumbs *b = container_of(work, typeof(*b), irq_work); |
| @@ -155,6 +167,8 @@ static void signal_irq_work(struct irq_work *work) · 7 unmodified lines | |||
| 155 | if (b->irq_armed && list_empty(&b->signalers)) | 167 | if (b->irq_armed && list_empty(&b->signalers)) |
| 156 | __intel_breadcrumbs_disarm_irq(b); | 168 | __intel_breadcrumbs_disarm_irq(b); |
| 157 | 169 | ||
| 170 | list_splice_init(&b->signaled_requests, &signal); | ||
| 171 | |||
| 158 | list_for_each_entry_safe(ce, cn, &b->signalers, signal_link) { | 172 | list_for_each_entry_safe(ce, cn, &b->signalers, signal_link) { |
| 159 | GEM_BUG_ON(list_empty(&ce->signals)); | 173 | GEM_BUG_ON(list_empty(&ce->signals)); |
| 160 | 174 | ||
| @@ -163,24 +177,15 @@ static void signal_irq_work(struct irq_work *work) · 2 unmodified lines | |||
| 163 | list_entry(pos, typeof(*rq), signal_link); | 177 | list_entry(pos, typeof(*rq), signal_link); |
| 164 | 178 | ||
| 165 | GEM_BUG_ON(!check_signal_order(ce, rq)); | 179 | GEM_BUG_ON(!check_signal_order(ce, rq)); |
| 166 | |||
| 167 | if (!__request_completed(rq)) | 180 | if (!__request_completed(rq)) |
| 168 | break; | 181 | break; |
| 169 | 182 | ||
| 170 | GEM_BUG_ON(!test_bit(I915_FENCE_FLAG_SIGNAL, | ||
| 171 | &rq->fence.flags)); | ||
| 172 | clear_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags); | ||
| 173 | |||
| 174 | if (!__dma_fence_signal(&rq->fence)) | ||
| 175 | continue; | ||
| 176 | |||
| 177 | /* | 183 | /* |
| 178 | * Queue for execution after dropping the signaling | 184 | * Queue for execution after dropping the signaling |
| 179 | * spinlock as the callback chain may end up adding | 185 | * spinlock as the callback chain may end up adding |
| 180 | * more signalers to the same context or engine. | 186 | * more signalers to the same context or engine. |
| 181 | */ | 187 | */ |
| 182 | i915_request_get(rq); | 188 | __signal_request(rq, &signal); |
| 183 | list_add_tail(&rq->signal_link, &signal); | ||
| 184 | } | 189 | } |
| 185 | 190 | ||
| 186 | /* | 191 | /* |
| @@ -255,6 +260,7 @@ void intel_engine_init_breadcrumbs(struct intel_engine_cs *engine) · 68 unmodified lines | |||
| 255 | 260 | ||
| 256 | spin_lock_init(&b->irq_lock); | 261 | spin_lock_init(&b->irq_lock); |
| 257 | INIT_LIST_HEAD(&b->signalers); | 262 | INIT_LIST_HEAD(&b->signalers); |
| 263 | INIT_LIST_HEAD(&b->signaled_requests); | ||
| 258 | 264 | ||
| 259 | init_irq_work(&b->irq_work, signal_irq_work); | 265 | init_irq_work(&b->irq_work, signal_irq_work); |
| 260 | } | 266 | } |
| @@ -274,6 +280,32 @@ void intel_engine_reset_breadcrumbs(struct intel_engine_cs *engine) · 13 unmodified lines | |||
| 274 | spin_unlock_irqrestore(&b->irq_lock, flags); | 280 | spin_unlock_irqrestore(&b->irq_lock, flags); |
| 275 | } | 281 | } |
| 276 | 282 | ||
| 283 | void intel_engine_transfer_stale_breadcrumbs(struct intel_engine_cs *engine, | ||
| 284 | struct intel_context *ce) | ||
| 285 | { | ||
| 286 | struct intel_breadcrumbs *b = &engine->breadcrumbs; | ||
| 287 | unsigned long flags; | ||
| 288 | |||
| 289 | spin_lock_irqsave(&b->irq_lock, flags); | ||
| 290 | if (!list_empty(&ce->signals)) { | ||
| 291 | struct i915_request *rq, *next; | ||
| 292 | |||
| 293 | /* Queue for executing the signal callbacks in the irq_work */ | ||
| 294 | list_for_each_entry_safe(rq, next, &ce->signals, signal_link) { | ||
| 295 | GEM_BUG_ON(rq->engine != engine); | ||
| 296 | GEM_BUG_ON(!__request_completed(rq)); | ||
| 297 | |||
| 298 | __signal_request(rq, &b->signaled_requests); | ||
| 299 | } | ||
| 300 | |||
| 301 | INIT_LIST_HEAD(&ce->signals); | ||
| 302 | list_del_init(&ce->signal_link); | ||
| 303 | |||
| 304 | irq_work_queue(&b->irq_work); | ||
| 305 | } | ||
| 306 | spin_unlock_irqrestore(&b->irq_lock, flags); | ||
| 307 | } | ||
| 308 | |||
| 277 | void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine) | 309 | void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine) |
| 278 | { | 310 | { |
| 279 | } | 311 | } |
| @@ -238,22 +238,35 @@ intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine) · 237 unmodified lines | |||
| 238 | void intel_engine_reset_breadcrumbs(struct intel_engine_cs *engine); | 238 | void intel_engine_reset_breadcrumbs(struct intel_engine_cs *engine); |
| 239 | void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine); | 239 | void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine); |
| 240 | 240 | ||
| 241 | void intel_engine_transfer_stale_breadcrumbs(struct intel_engine_cs *engine, | ||
| 242 | struct intel_context *ce); | ||
| 243 | |||
| 241 | void intel_engine_print_breadcrumbs(struct intel_engine_cs *engine, | 244 | void intel_engine_print_breadcrumbs(struct intel_engine_cs *engine, |
| 242 | struct drm_printer *p); | 245 | struct drm_printer *p); |
| 243 | 246 | ||
| 244 | static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset) | 247 | static inline u32 *__gen8_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset) |
| 245 | { | 248 | { |
| 246 | memset(batch, 0, 6 * sizeof(u32)); | 249 | memset(batch, 0, 6 * sizeof(u32)); |
| 247 | 250 | ||
| 248 | batch[0] = GFX_OP_PIPE_CONTROL(6); | 251 | batch[0] = GFX_OP_PIPE_CONTROL(6) | flags0; |
| 249 | batch[1] = flags; | 252 | batch[1] = flags1; |
| 250 | batch[2] = offset; | 253 | batch[2] = offset; |
| 251 | 254 | ||
| 252 | return batch + 6; | 255 | return batch + 6; |
| 253 | } | 256 | } |
| 254 | 257 | ||
| 258 | static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset) | ||
| 259 | { | ||
| 260 | return __gen8_emit_pipe_control(batch, 0, flags, offset); | ||
| 261 | } | ||
| 262 | |||
| 263 | static inline u32 *gen12_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset) | ||
| 264 | { | ||
| 265 | return __gen8_emit_pipe_control(batch, flags0, flags1, offset); | ||
| 266 | } | ||
| 267 | |||
| 255 | static inline u32 * | 268 | static inline u32 * |
| 256 | gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags) | 269 | __gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1) |
| 257 | { | 270 | { |
| 258 | /* We're using qword write, offset should be aligned to 8 bytes. */ | 271 | /* We're using qword write, offset should be aligned to 8 bytes. */ |
| 259 | GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8)); | 272 | GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8)); |
| @@ -262,8 +275,8 @@ gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags) · 2 unmodified lines | |||
| 262 | * need a prior CS_STALL, which is emitted by the flush | 275 | * need a prior CS_STALL, which is emitted by the flush |
| 263 | * following the batch. | 276 | * following the batch. |
| 264 | */ | 277 | */ |
| 265 | *cs++ = GFX_OP_PIPE_CONTROL(6); | 278 | *cs++ = GFX_OP_PIPE_CONTROL(6) | flags0; |
| 266 | *cs++ = flags | PIPE_CONTROL_QW_WRITE | PIPE_CONTROL_GLOBAL_GTT_IVB; | 279 | *cs++ = flags1 | PIPE_CONTROL_QW_WRITE | PIPE_CONTROL_GLOBAL_GTT_IVB; |
| 267 | *cs++ = gtt_offset; | 280 | *cs++ = gtt_offset; |
| 268 | *cs++ = 0; | 281 | *cs++ = 0; |
| 269 | *cs++ = value; | 282 | *cs++ = value; |
| @@ -273,6 +286,18 @@ gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags) · 3 unmodified lines | |||
| 273 | return cs; | 286 | return cs; |
| 274 | } | 287 | } |
| 275 | 288 | ||
| 289 | static inline u32* | ||
| 290 | gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags) | ||
| 291 | { | ||
| 292 | return __gen8_emit_ggtt_write_rcs(cs, value, gtt_offset, 0, flags); | ||
| 293 | } | ||
| 294 | |||
| 295 | static inline u32* | ||
| 296 | gen12_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1) | ||
| 297 | { | ||
| 298 | return __gen8_emit_ggtt_write_rcs(cs, value, gtt_offset, flags0, flags1); | ||
| 299 | } | ||
| 300 | |||
| 276 | static inline u32 * | 301 | static inline u32 * |
| 277 | gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags) | 302 | gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags) |
| 278 | { | 303 | { |
| @@ -332,13 +357,4 @@ intel_engine_has_preempt_reset(const struct intel_engine_cs *engine) · 53 unmodified lines | |||
| 332 | return intel_engine_has_preemption(engine); | 357 | return intel_engine_has_preemption(engine); |
| 333 | } | 358 | } |
| 334 | 359 | ||
| 335 | static inline bool | ||
| 336 | intel_engine_has_timeslices(const struct intel_engine_cs *engine) | ||
| 337 | { | ||
| 338 | if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION)) | ||
| 339 | return false; | ||
| 340 | |||
| 341 | return intel_engine_has_semaphores(engine); | ||
| 342 | } | ||
| 343 | |||
| 344 | #endif /* _INTEL_RINGBUFFER_H_ */ | 360 | #endif /* _INTEL_RINGBUFFER_H_ */ |
| @@ -31,7 +31,6 @@ · 30 unmodified lines | |||
| 31 | #include "intel_context.h" | 31 | #include "intel_context.h" |
| 32 | #include "intel_engine.h" | 32 | #include "intel_engine.h" |
| 33 | #include "intel_engine_pm.h" | 33 | #include "intel_engine_pm.h" |
| 34 | #include "intel_engine_pool.h" | ||
| 35 | #include "intel_engine_user.h" | 34 | #include "intel_engine_user.h" |
| 36 | #include "intel_gt.h" | 35 | #include "intel_gt.h" |
| 37 | #include "intel_gt_requests.h" | 36 | #include "intel_gt_requests.h" |
| @@ -327,6 +326,8 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id) · 289 unmodified lines | |||
| 327 | if (INTEL_GEN(i915) == 12 && engine->class == RENDER_CLASS) | 326 | if (INTEL_GEN(i915) == 12 && engine->class == RENDER_CLASS) |
| 328 | engine->props.preempt_timeout_ms = 0; | 327 | engine->props.preempt_timeout_ms = 0; |
| 329 | 328 | ||
| 329 | engine->defaults = engine->props; /* never to change again */ | ||
| 330 | |||
| 330 | engine->context_size = intel_engine_context_size(gt, engine->class); | 331 | engine->context_size = intel_engine_context_size(gt, engine->class); |
| 331 | if (WARN_ON(engine->context_size > BIT(20))) | 332 | if (WARN_ON(engine->context_size > BIT(20))) |
| 332 | engine->context_size = 0; | 333 | engine->context_size = 0; |
| @@ -631,8 +632,6 @@ static int engine_setup_common(struct intel_engine_cs *engine) · 298 unmodified lines | |||
| 631 | intel_engine_init__pm(engine); | 632 | intel_engine_init__pm(engine); |
| 632 | intel_engine_init_retire(engine); | 633 | intel_engine_init_retire(engine); |
| 633 | 634 | ||
| 634 | intel_engine_pool_init(&engine->pool); | ||
| 635 | |||
| 636 | /* Use the whole device by default */ | 635 | /* Use the whole device by default */ |
| 637 | engine->sseu = | 636 | engine->sseu = |
| 638 | intel_sseu_from_device_info(&RUNTIME_INFO(engine->i915)->sseu); | 637 | intel_sseu_from_device_info(&RUNTIME_INFO(engine->i915)->sseu); |
| @@ -829,7 +828,6 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine) · 190 unmodified lines | |||
| 829 | cleanup_status_page(engine); | 828 | cleanup_status_page(engine); |
| 830 | 829 | ||
| 831 | intel_engine_fini_retire(engine); | 830 | intel_engine_fini_retire(engine); |
| 832 | intel_engine_pool_fini(&engine->pool); | ||
| 833 | intel_engine_fini_breadcrumbs(engine); | 831 | intel_engine_fini_breadcrumbs(engine); |
| 834 | intel_engine_cleanup_cmd_parser(engine); | 832 | intel_engine_cleanup_cmd_parser(engine); |
| 835 | 833 | ||
| @@ -10,7 +10,6 @@ · 9 unmodified lines | |||
| 10 | #include "intel_engine.h" | 10 | #include "intel_engine.h" |
| 11 | #include "intel_engine_heartbeat.h" | 11 | #include "intel_engine_heartbeat.h" |
| 12 | #include "intel_engine_pm.h" | 12 | #include "intel_engine_pm.h" |
| 13 | #include "intel_engine_pool.h" | ||
| 14 | #include "intel_gt.h" | 13 | #include "intel_gt.h" |
| 15 | #include "intel_gt_pm.h" | 14 | #include "intel_gt_pm.h" |
| 16 | #include "intel_rc6.h" | 15 | #include "intel_rc6.h" |
| @@ -22,18 +21,11 @@ static int __engine_unpark(struct intel_wakeref *wf) · 5 unmodified lines | |||
| 22 | struct intel_engine_cs *engine = | 21 | struct intel_engine_cs *engine = |
| 23 | container_of(wf, typeof(*engine), wakeref); | 22 | container_of(wf, typeof(*engine), wakeref); |
| 24 | struct intel_context *ce; | 23 | struct intel_context *ce; |
| 25 | void *map; | ||
| 26 | 24 | ||
| 27 | ENGINE_TRACE(engine, "\n"); | 25 | ENGINE_TRACE(engine, "\n"); |
| 28 | 26 | ||
| 29 | intel_gt_pm_get(engine->gt); | 27 | intel_gt_pm_get(engine->gt); |
| 30 | 28 | ||
| 31 | /* Pin the default state for fast resets from atomic context. */ | ||
| 32 | map = NULL; | ||
| 33 | if (engine->default_state) | ||
| 34 | map = shmem_pin_map(engine->default_state); | ||
| 35 | engine->pinned_default_state = map; | ||
| 36 | |||
| 37 | /* Discard stale context state from across idling */ | 29 | /* Discard stale context state from across idling */ |
| 38 | ce = engine->kernel_context; | 30 | ce = engine->kernel_context; |
| 39 | if (ce) { | 31 | if (ce) { |
| @@ -43,6 +35,7 @@ static int __engine_unpark(struct intel_wakeref *wf) · 3 unmodified lines | |||
| 43 | if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM) && ce->state) { | 35 | if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM) && ce->state) { |
| 44 | struct drm_i915_gem_object *obj = ce->state->obj; | 36 | struct drm_i915_gem_object *obj = ce->state->obj; |
| 45 | int type = i915_coherent_map_type(engine->i915); | 37 | int type = i915_coherent_map_type(engine->i915); |
| 38 | void *map; | ||
| 46 | 39 | ||
| 47 | map = i915_gem_object_pin_map(obj, type); | 40 | map = i915_gem_object_pin_map(obj, type); |
| 48 | if (!IS_ERR(map)) { | 41 | if (!IS_ERR(map)) { |
| @@ -254,7 +247,6 @@ static int __engine_park(struct intel_wakeref *wf) · 205 unmodified lines | |||
| 254 | 247 | ||
| 255 | intel_engine_park_heartbeat(engine); | 248 | intel_engine_park_heartbeat(engine); |
| 256 | intel_engine_disarm_breadcrumbs(engine); | 249 | intel_engine_disarm_breadcrumbs(engine); |
| 257 | intel_engine_pool_park(&engine->pool); | ||
| 258 | 250 | ||
| 259 | /* Must be reset upon idling, or we may miss the busy wakeup. */ | 251 | /* Must be reset upon idling, or we may miss the busy wakeup. */ |
| 260 | GEM_BUG_ON(engine->execlists.queue_priority_hint != INT_MIN); | 252 | GEM_BUG_ON(engine->execlists.queue_priority_hint != INT_MIN); |
| @@ -262,12 +254,6 @@ static int __engine_park(struct intel_wakeref *wf) · 1 unmodified line | |||
| 262 | if (engine->park) | 254 | if (engine->park) |
| 263 | engine->park(engine); | 255 | engine->park(engine); |
| 264 | 256 | ||
| 265 | if (engine->pinned_default_state) { | ||
| 266 | shmem_unpin_map(engine->default_state, | ||
| 267 | engine->pinned_default_state); | ||
| 268 | engine->pinned_default_state = NULL; | ||
| 269 | } | ||
| 270 | |||
| 271 | engine->execlists.no_priolist = false; | 257 | engine->execlists.no_priolist = false; |
| 272 | 258 | ||
| 273 | /* While gt calls i915_vma_parked(), we have to break the lock cycle */ | 259 | /* While gt calls i915_vma_parked(), we have to break the lock cycle */ |
| @@ -1,34 +0,0 @@ | |||
| 1 | /* | ||
| 2 | * SPDX-License-Identifier: MIT | ||
| 3 | * | ||
| 4 | * Copyright © 2014-2018 Intel Corporation | ||
| 5 | */ | ||
| 6 | |||
| 7 | #ifndef INTEL_ENGINE_POOL_H | ||
| 8 | #define INTEL_ENGINE_POOL_H | ||
| 9 | |||
| 10 | #include "intel_engine_pool_types.h" | ||
| 11 | #include "i915_active.h" | ||
| 12 | #include "i915_request.h" | ||
| 13 | |||
| 14 | struct intel_engine_pool_node * | ||
| 15 | intel_engine_get_pool(struct intel_engine_cs *engine, size_t size); | ||
| 16 | |||
| 17 | static inline int | ||
| 18 | intel_engine_pool_mark_active(struct intel_engine_pool_node *node, | ||
| 19 | struct i915_request *rq) | ||
| 20 | { | ||
| 21 | return i915_active_add_request(&node->active, rq); | ||
| 22 | } | ||
| 23 | |||
| 24 | static inline void | ||
| 25 | intel_engine_pool_put(struct intel_engine_pool_node *node) | ||
| 26 | { | ||
| 27 | i915_active_release(&node->active); | ||
| 28 | } | ||
| 29 | |||
| 30 | void intel_engine_pool_init(struct intel_engine_pool *pool); | ||
| 31 | void intel_engine_pool_park(struct intel_engine_pool *pool); | ||
| 32 | void intel_engine_pool_fini(struct intel_engine_pool *pool); | ||
| 33 | |||
| 34 | #endif /* INTEL_ENGINE_POOL_H */ | ||
| @@ -22,7 +22,6 @@ · 21 unmodified lines | |||
| 22 | #include "i915_pmu.h" | 22 | #include "i915_pmu.h" |
| 23 | #include "i915_priolist_types.h" | 23 | #include "i915_priolist_types.h" |
| 24 | #include "i915_selftest.h" | 24 | #include "i915_selftest.h" |
| 25 | #include "intel_engine_pool_types.h" | ||
| 26 | #include "intel_sseu.h" | 25 | #include "intel_sseu.h" |
| 27 | #include "intel_timeline_types.h" | 26 | #include "intel_timeline_types.h" |
| 28 | #include "intel_wakeref.h" | 27 | #include "intel_wakeref.h" |
| @@ -180,6 +179,11 @@ struct intel_engine_execlists { · 151 unmodified lines | |||
| 180 | */ | 179 | */ |
| 181 | u32 error_interrupt; | 180 | u32 error_interrupt; |
| 182 | 181 | ||
| 182 | /** | ||
| 183 | * @reset_ccid: Active CCID [EXECLISTS_STATUS_HI] at the time of reset | ||
| 184 | */ | ||
| 185 | u32 reset_ccid; | ||
| 186 | |||
| 183 | /** | 187 | /** |
| 184 | * @no_priolist: priority lists disabled | 188 | * @no_priolist: priority lists disabled |
| 185 | */ | 189 | */ |
| @@ -340,7 +344,6 @@ struct intel_engine_cs { · 154 unmodified lines | |||
| 340 | unsigned long wakeref_serial; | 344 | unsigned long wakeref_serial; |
| 341 | struct intel_wakeref wakeref; | 345 | struct intel_wakeref wakeref; |
| 342 | struct file *default_state; | 346 | struct file *default_state; |
| 343 | void *pinned_default_state; | ||
| 344 | 347 | ||
| 345 | struct { | 348 | struct { |
| 346 | struct intel_ring *ring; | 349 | struct intel_ring *ring; |
| @@ -374,6 +377,8 @@ struct intel_engine_cs { · 27 unmodified lines | |||
| 374 | spinlock_t irq_lock; | 377 | spinlock_t irq_lock; |
| 375 | struct list_head signalers; | 378 | struct list_head signalers; |
| 376 | 379 | ||
| 380 | struct list_head signaled_requests; | ||
| 381 | |||
| 377 | struct irq_work irq_work; /* for use from inside irq_lock */ | 382 | struct irq_work irq_work; /* for use from inside irq_lock */ |
| 378 | 383 | ||
| 379 | unsigned int irq_enabled; | 384 | unsigned int irq_enabled; |
| @@ -405,13 +410,6 @@ struct intel_engine_cs { · 25 unmodified lines | |||
| 405 | struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_COUNT]; | 410 | struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_COUNT]; |
| 406 | } pmu; | 411 | } pmu; |
| 407 | 412 | ||
| 408 | /* | ||
| 409 | * A pool of objects to use as shadow copies of client batch buffers | ||
| 410 | * when the command parser is enabled. Prevents the client from | ||
| 411 | * modifying the batch contents after software parsing. | ||
| 412 | */ | ||
| 413 | struct intel_engine_pool pool; | ||
| 414 | |||
| 415 | struct intel_hw_status_page status_page; | 413 | struct intel_hw_status_page status_page; |
| 416 | struct i915_ctx_workarounds wa_ctx; | 414 | struct i915_ctx_workarounds wa_ctx; |
| 417 | struct i915_wa_list ctx_wa_list; | 415 | struct i915_wa_list ctx_wa_list; |
| @@ -500,10 +498,11 @@ struct intel_engine_cs { · 82 unmodified lines | |||
| 500 | #define I915_ENGINE_SUPPORTS_STATS BIT(1) | 498 | #define I915_ENGINE_SUPPORTS_STATS BIT(1) |
| 501 | #define I915_ENGINE_HAS_PREEMPTION BIT(2) | 499 | #define I915_ENGINE_HAS_PREEMPTION BIT(2) |
| 502 | #define I915_ENGINE_HAS_SEMAPHORES BIT(3) | 500 | #define I915_ENGINE_HAS_SEMAPHORES BIT(3) |
| 503 | #define I915_ENGINE_NEEDS_BREADCRUMB_TASKLET BIT(4) | 501 | #define I915_ENGINE_HAS_TIMESLICES BIT(4) |
| 504 | #define I915_ENGINE_IS_VIRTUAL BIT(5) | 502 | #define I915_ENGINE_NEEDS_BREADCRUMB_TASKLET BIT(5) |
| 505 | #define I915_ENGINE_HAS_RELATIVE_MMIO BIT(6) | 503 | #define I915_ENGINE_IS_VIRTUAL BIT(6) |
| 506 | #define I915_ENGINE_REQUIRES_CMD_PARSER BIT(7) | 504 | #define I915_ENGINE_HAS_RELATIVE_MMIO BIT(7) |
| 505 | #define I915_ENGINE_REQUIRES_CMD_PARSER BIT(8) | ||
| 507 | unsigned int flags; | 506 | unsigned int flags; |
| 508 | 507 | ||
| 509 | /* | 508 | /* |
| @@ -568,7 +567,7 @@ struct intel_engine_cs { · 58 unmodified lines | |||
| 568 | unsigned long preempt_timeout_ms; | 567 | unsigned long preempt_timeout_ms; |
| 569 | unsigned long stop_timeout_ms; | 568 | unsigned long stop_timeout_ms; |
| 570 | unsigned long timeslice_duration_ms; | 569 | unsigned long timeslice_duration_ms; |
| 571 | } props; | 570 | } props, defaults; |
| 572 | }; | 571 | }; |
| 573 | 572 | ||
| 574 | static inline bool | 573 | static inline bool |
| @@ -601,6 +600,15 @@ intel_engine_has_semaphores(const struct intel_engine_cs *engine) · 26 unmodified lines | |||
| 601 | return engine->flags & I915_ENGINE_HAS_SEMAPHORES; | 600 | return engine->flags & I915_ENGINE_HAS_SEMAPHORES; |
| 602 | } | 601 | } |
| 603 | 602 | ||
| 603 | static inline bool | ||
| 604 | intel_engine_has_timeslices(const struct intel_engine_cs *engine) | ||
| 605 | { | ||
| 606 | if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION)) | ||
| 607 | return false; | ||
| 608 | |||
| 609 | return engine->flags & I915_ENGINE_HAS_TIMESLICES; | ||
| 610 | } | ||
| 611 | |||
| 604 | static inline bool | 612 | static inline bool |
| 605 | intel_engine_needs_breadcrumb_tasklet(const struct intel_engine_cs *engine) | 613 | intel_engine_needs_breadcrumb_tasklet(const struct intel_engine_cs *engine) |
| 606 | { | 614 | { |
| @@ -236,9 +236,8 @@ · 235 unmodified lines | |||
| 236 | #define PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH (1<<12) /* gen6+ */ | 236 | #define PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH (1<<12) /* gen6+ */ |
| 237 | #define PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE (1<<11) /* MBZ on ILK */ | 237 | #define PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE (1<<11) /* MBZ on ILK */ |
| 238 | #define PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE (1<<10) /* GM45+ only */ | 238 | #define PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE (1<<10) /* GM45+ only */ |
| 239 | #define PIPE_CONTROL_L3_RO_CACHE_INVALIDATE REG_BIT(10) /* gen12 */ | ||
| 240 | #define PIPE_CONTROL_INDIRECT_STATE_DISABLE (1<<9) | 239 | #define PIPE_CONTROL_INDIRECT_STATE_DISABLE (1<<9) |
| 241 | #define PIPE_CONTROL_HDC_PIPELINE_FLUSH REG_BIT(9) /* gen12 */ | 240 | #define PIPE_CONTROL0_HDC_PIPELINE_FLUSH REG_BIT(9) /* gen12 */ |
| 242 | #define PIPE_CONTROL_NOTIFY (1<<8) | 241 | #define PIPE_CONTROL_NOTIFY (1<<8) |
| 243 | #define PIPE_CONTROL_FLUSH_ENABLE (1<<7) /* gen7+ */ | 242 | #define PIPE_CONTROL_FLUSH_ENABLE (1<<7) /* gen7+ */ |
| 244 | #define PIPE_CONTROL_DC_FLUSH_ENABLE (1<<5) | 243 | #define PIPE_CONTROL_DC_FLUSH_ENABLE (1<<5) |
| @@ -7,6 +7,7 @@ · 6 unmodified lines | |||
| 7 | #include "i915_drv.h" | 7 | #include "i915_drv.h" |
| 8 | #include "intel_context.h" | 8 | #include "intel_context.h" |
| 9 | #include "intel_gt.h" | 9 | #include "intel_gt.h" |
| 10 | #include "intel_gt_buffer_pool.h" | ||
| 10 | #include "intel_gt_clock_utils.h" | 11 | #include "intel_gt_clock_utils.h" |
| 11 | #include "intel_gt_pm.h" | 12 | #include "intel_gt_pm.h" |
| 12 | #include "intel_gt_requests.h" | 13 | #include "intel_gt_requests.h" |
| @@ -28,6 +29,7 @@ void intel_gt_init_early(struct intel_gt *gt, struct drm_i915_private *i915) · 15 unmodified lines | |||
| 28 | INIT_LIST_HEAD(>->closed_vma); | 29 | INIT_LIST_HEAD(>->closed_vma); |
| 29 | spin_lock_init(>->closed_lock); | 30 | spin_lock_init(>->closed_lock); |
| 30 | 31 | ||
| 32 | intel_gt_init_buffer_pool(gt); | ||
| 31 | intel_gt_init_reset(gt); | 33 | intel_gt_init_reset(gt); |
| 32 | intel_gt_init_requests(gt); | 34 | intel_gt_init_requests(gt); |
| 33 | intel_gt_init_timelines(gt); | 35 | intel_gt_init_timelines(gt); |
| @@ -621,6 +623,7 @@ void intel_gt_driver_release(struct intel_gt *gt) · 587 unmodified lines | |||
| 621 | 623 | ||
| 622 | intel_gt_pm_fini(gt); | 624 | intel_gt_pm_fini(gt); |
| 623 | intel_gt_fini_scratch(gt); | 625 | intel_gt_fini_scratch(gt); |
| 626 | intel_gt_fini_buffer_pool(gt); | ||
| 624 | } | 627 | } |
| 625 | 628 | ||
| 626 | void intel_gt_driver_late_release(struct intel_gt *gt) | 629 | void intel_gt_driver_late_release(struct intel_gt *gt) |
| @@ -1,6 +1,5 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 1 | /* | 2 | /* |
| 2 | * SPDX-License-Identifier: MIT | ||
| 3 | * | ||
| 4 | * Copyright © 2014-2018 Intel Corporation | 3 | * Copyright © 2014-2018 Intel Corporation |
| 5 | */ | 4 | */ |
| 6 | 5 | ||
| @@ -8,15 +7,15 @@ · 1 unmodified line | |||
| 8 | 7 | ||
| 9 | #include "i915_drv.h" | 8 | #include "i915_drv.h" |
| 10 | #include "intel_engine_pm.h" | 9 | #include "intel_engine_pm.h" |
| 11 | #include "intel_engine_pool.h" | 10 | #include "intel_gt_buffer_pool.h" |
| 12 | 11 | ||
| 13 | static struct intel_engine_cs *to_engine(struct intel_engine_pool *pool) | 12 | static struct intel_gt *to_gt(struct intel_gt_buffer_pool *pool) |
| 14 | { | 13 | { |
| 15 | return container_of(pool, struct intel_engine_cs, pool); | 14 | return container_of(pool, struct intel_gt, buffer_pool); |
| 16 | } | 15 | } |
| 17 | 16 | ||
| 18 | static struct list_head * | 17 | static struct list_head * |
| 19 | bucket_for_size(struct intel_engine_pool *pool, size_t sz) | 18 | bucket_for_size(struct intel_gt_buffer_pool *pool, size_t sz) |
| 20 | { | 19 | { |
| 21 | int n; | 20 | int n; |
| 22 | 21 | ||
| @@ -32,16 +31,50 @@ bucket_for_size(struct intel_engine_pool *pool, size_t sz) · 9 unmodified lines | |||
| 32 | return &pool->cache_list[n]; | 31 | return &pool->cache_list[n]; |
| 33 | } | 32 | } |
| 34 | 33 | ||
| 35 | static void node_free(struct intel_engine_pool_node *node) | 34 | static void node_free(struct intel_gt_buffer_pool_node *node) |
| 36 | { | 35 | { |
| 37 | i915_gem_object_put(node->obj); | 36 | i915_gem_object_put(node->obj); |
| 38 | i915_active_fini(&node->active); | 37 | i915_active_fini(&node->active); |
| 39 | kfree(node); | 38 | kfree(node); |
| 40 | } | 39 | } |
| 41 | 40 | ||
| 41 | static void pool_free_work(struct work_struct *wrk) | ||
| 42 | { | ||
| 43 | struct intel_gt_buffer_pool *pool = | ||
| 44 | container_of(wrk, typeof(*pool), work.work); | ||
| 45 | struct intel_gt_buffer_pool_node *node, *next; | ||
| 46 | unsigned long old = jiffies - HZ; | ||
| 47 | bool active = false; | ||
| 48 | LIST_HEAD(stale); | ||
| 49 | int n; | ||
| 50 | |||
| 51 | /* Free buffers that have not been used in the past second */ | ||
| 52 | spin_lock_irq(&pool->lock); | ||
| 53 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) { | ||
| 54 | struct list_head *list = &pool->cache_list[n]; | ||
| 55 | |||
| 56 | /* Most recent at head; oldest at tail */ | ||
| 57 | list_for_each_entry_safe_reverse(node, next, list, link) { | ||
| 58 | if (time_before(node->age, old)) | ||
| 59 | break; | ||
| 60 | |||
| 61 | list_move(&node->link, &stale); | ||
| 62 | } | ||
| 63 | active |= !list_empty(list); | ||
| 64 | } | ||
| 65 | spin_unlock_irq(&pool->lock); | ||
| 66 | |||
| 67 | list_for_each_entry_safe(node, next, &stale, link) | ||
| 68 | node_free(node); | ||
| 69 | |||
| 70 | if (active) | ||
| 71 | schedule_delayed_work(&pool->work, | ||
| 72 | round_jiffies_up_relative(HZ)); | ||
| 73 | } | ||
| 74 | |||
| 42 | static int pool_active(struct i915_active *ref) | 75 | static int pool_active(struct i915_active *ref) |
| 43 | { | 76 | { |
| 44 | struct intel_engine_pool_node *node = | 77 | struct intel_gt_buffer_pool_node *node = |
| 45 | container_of(ref, typeof(*node), active); | 78 | container_of(ref, typeof(*node), active); |
| 46 | struct dma_resv *resv = node->obj->base.resv; | 79 | struct dma_resv *resv = node->obj->base.resv; |
| 47 | int err; | 80 | int err; |
| @@ -64,29 +97,31 @@ static int pool_active(struct i915_active *ref) · 16 unmodified lines | |||
| 64 | __i915_active_call | 97 | __i915_active_call |
| 65 | static void pool_retire(struct i915_active *ref) | 98 | static void pool_retire(struct i915_active *ref) |
| 66 | { | 99 | { |
| 67 | struct intel_engine_pool_node *node = | 100 | struct intel_gt_buffer_pool_node *node = |
| 68 | container_of(ref, typeof(*node), active); | 101 | container_of(ref, typeof(*node), active); |
| 69 | struct intel_engine_pool *pool = node->pool; | 102 | struct intel_gt_buffer_pool *pool = node->pool; |
| 70 | struct list_head *list = bucket_for_size(pool, node->obj->base.size); | 103 | struct list_head *list = bucket_for_size(pool, node->obj->base.size); |
| 71 | unsigned long flags; | 104 | unsigned long flags; |
| 72 | 105 | ||
| 73 | GEM_BUG_ON(!intel_engine_pm_is_awake(to_engine(pool))); | ||
| 74 | |||
| 75 | i915_gem_object_unpin_pages(node->obj); | 106 | i915_gem_object_unpin_pages(node->obj); |
| 76 | 107 | ||
| 77 | /* Return this object to the shrinker pool */ | 108 | /* Return this object to the shrinker pool */ |
| 78 | i915_gem_object_make_purgeable(node->obj); | 109 | i915_gem_object_make_purgeable(node->obj); |
| 79 | 110 | ||
| 80 | spin_lock_irqsave(&pool->lock, flags); | 111 | spin_lock_irqsave(&pool->lock, flags); |
| 112 | node->age = jiffies; | ||
| 81 | list_add(&node->link, list); | 113 | list_add(&node->link, list); |
| 82 | spin_unlock_irqrestore(&pool->lock, flags); | 114 | spin_unlock_irqrestore(&pool->lock, flags); |
| 115 | |||
| 116 | schedule_delayed_work(&pool->work, | ||
| 117 | round_jiffies_up_relative(HZ)); | ||
| 83 | } | 118 | } |
| 84 | 119 | ||
| 85 | static struct intel_engine_pool_node * | 120 | static struct intel_gt_buffer_pool_node * |
| 86 | node_create(struct intel_engine_pool *pool, size_t sz) | 121 | node_create(struct intel_gt_buffer_pool *pool, size_t sz) |
| 87 | { | 122 | { |
| 88 | struct intel_engine_cs *engine = to_engine(pool); | 123 | struct intel_gt *gt = to_gt(pool); |
| 89 | struct intel_engine_pool_node *node; | 124 | struct intel_gt_buffer_pool_node *node; |
| 90 | struct drm_i915_gem_object *obj; | 125 | struct drm_i915_gem_object *obj; |
| 91 | 126 | ||
| 92 | node = kmalloc(sizeof(*node), | 127 | node = kmalloc(sizeof(*node), |
| @@ -97,7 +132,7 @@ node_create(struct intel_engine_pool *pool, size_t sz) · 4 unmodified lines | |||
| 97 | node->pool = pool; | 132 | node->pool = pool; |
| 98 | i915_active_init(&node->active, pool_active, pool_retire); | 133 | i915_active_init(&node->active, pool_active, pool_retire); |
| 99 | 134 | ||
| 100 | obj = i915_gem_object_create_internal(engine->i915, sz); | 135 | obj = i915_gem_object_create_internal(gt->i915, sz); |
| 101 | if (IS_ERR(obj)) { | 136 | if (IS_ERR(obj)) { |
| 102 | i915_active_fini(&node->active); | 137 | i915_active_fini(&node->active); |
| 103 | kfree(node); | 138 | kfree(node); |
| @@ -110,26 +145,15 @@ node_create(struct intel_engine_pool *pool, size_t sz) · 6 unmodified lines | |||
| 110 | return node; | 145 | return node; |
| 111 | } | 146 | } |
| 112 | 147 | ||
| 113 | static struct intel_engine_pool *lookup_pool(struct intel_engine_cs *engine) | 148 | struct intel_gt_buffer_pool_node * |
| 149 | intel_gt_get_buffer_pool(struct intel_gt *gt, size_t size) | ||
| 114 | { | 150 | { |
| 115 | if (intel_engine_is_virtual(engine)) | 151 | struct intel_gt_buffer_pool *pool = >->buffer_pool; |
| 116 | engine = intel_virtual_engine_get_sibling(engine, 0); | 152 | struct intel_gt_buffer_pool_node *node; |
| 117 | |||
| 118 | GEM_BUG_ON(!engine); | ||
| 119 | return &engine->pool; | ||
| 120 | } | ||
| 121 | |||
| 122 | struct intel_engine_pool_node * | ||
| 123 | intel_engine_get_pool(struct intel_engine_cs *engine, size_t size) | ||
| 124 | { | ||
| 125 | struct intel_engine_pool *pool = lookup_pool(engine); | ||
| 126 | struct intel_engine_pool_node *node; | ||
| 127 | struct list_head *list; | 153 | struct list_head *list; |
| 128 | unsigned long flags; | 154 | unsigned long flags; |
| 129 | int ret; | 155 | int ret; |
| 130 | 156 | ||
| 131 | GEM_BUG_ON(!intel_engine_pm_is_awake(to_engine(pool))); | ||
| 132 | |||
| 133 | size = PAGE_ALIGN(size); | 157 | size = PAGE_ALIGN(size); |
| 134 | list = bucket_for_size(pool, size); | 158 | list = bucket_for_size(pool, size); |
| 135 | 159 | ||
| @@ -157,34 +181,48 @@ intel_engine_get_pool(struct intel_engine_cs *engine, size_t size) · 21 unmodified lines | |||
| 157 | return node; | 181 | return node; |
| 158 | } | 182 | } |
| 159 | 183 | ||
| 160 | void intel_engine_pool_init(struct intel_engine_pool *pool) | 184 | void intel_gt_init_buffer_pool(struct intel_gt *gt) |
| 161 | { | 185 | { |
| 186 | struct intel_gt_buffer_pool *pool = >->buffer_pool; | ||
| 162 | int n; | 187 | int n; |
| 163 | 188 | ||
| 164 | spin_lock_init(&pool->lock); | 189 | spin_lock_init(&pool->lock); |
| 165 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) | 190 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) |
| 166 | INIT_LIST_HEAD(&pool->cache_list[n]); | 191 | INIT_LIST_HEAD(&pool->cache_list[n]); |
| 192 | INIT_DELAYED_WORK(&pool->work, pool_free_work); | ||
| 167 | } | 193 | } |
| 168 | 194 | ||
| 169 | void intel_engine_pool_park(struct intel_engine_pool *pool) | 195 | static void pool_free_imm(struct intel_gt_buffer_pool *pool) |
| 170 | { | 196 | { |
| 171 | int n; | 197 | int n; |
| 172 | 198 | ||
| 199 | spin_lock_irq(&pool->lock); | ||
| 173 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) { | 200 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) { |
| 201 | struct intel_gt_buffer_pool_node *node, *next; | ||
| 174 | struct list_head *list = &pool->cache_list[n]; | 202 | struct list_head *list = &pool->cache_list[n]; |
| 175 | struct intel_engine_pool_node *node, *nn; | ||
| 176 | 203 | ||
| 177 | list_for_each_entry_safe(node, nn, list, link) | 204 | list_for_each_entry_safe(node, next, list, link) |
| 178 | node_free(node); | 205 | node_free(node); |
| 179 | |||
| 180 | INIT_LIST_HEAD(list); | 206 | INIT_LIST_HEAD(list); |
| 181 | } | 207 | } |
| 208 | spin_unlock_irq(&pool->lock); | ||
| 209 | } | ||
| 210 | |||
| 211 | void intel_gt_flush_buffer_pool(struct intel_gt *gt) | ||
| 212 | { | ||
| 213 | struct intel_gt_buffer_pool *pool = >->buffer_pool; | ||
| 214 | |||
| 215 | if (cancel_delayed_work_sync(&pool->work)) | ||
| 216 | pool_free_imm(pool); | ||
| 182 | } | 217 | } |
| 183 | 218 | ||
| 184 | void intel_engine_pool_fini(struct intel_engine_pool *pool) | 219 | void intel_gt_fini_buffer_pool(struct intel_gt *gt) |
| 185 | { | 220 | { |
| 221 | struct intel_gt_buffer_pool *pool = >->buffer_pool; | ||
| 186 | int n; | 222 | int n; |
| 187 | 223 | ||
| 224 | intel_gt_flush_buffer_pool(gt); | ||
| 225 | |||
| 188 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) | 226 | for (n = 0; n < ARRAY_SIZE(pool->cache_list); n++) |
| 189 | GEM_BUG_ON(!list_empty(&pool->cache_list[n])); | 227 | GEM_BUG_ON(!list_empty(&pool->cache_list[n])); |
| 190 | } | 228 | } |
| @@ -0,0 +1,37 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | ||
| 2 | /* | ||
| 3 | * Copyright © 2014-2018 Intel Corporation | ||
| 4 | */ | ||
| 5 | |||
| 6 | #ifndef INTEL_GT_BUFFER_POOL_H | ||
| 7 | #define INTEL_GT_BUFFER_POOL_H | ||
| 8 | |||
| 9 | #include <linux/types.h> | ||
| 10 | |||
| 11 | #include "i915_active.h" | ||
| 12 | #include "intel_gt_buffer_pool_types.h" | ||
| 13 | |||
| 14 | struct intel_gt; | ||
| 15 | struct i915_request; | ||
| 16 | |||
| 17 | struct intel_gt_buffer_pool_node * | ||
| 18 | intel_gt_get_buffer_pool(struct intel_gt *gt, size_t size); | ||
| 19 | |||
| 20 | static inline int | ||
| 21 | intel_gt_buffer_pool_mark_active(struct intel_gt_buffer_pool_node *node, | ||
| 22 | struct i915_request *rq) | ||
| 23 | { | ||
| 24 | return i915_active_add_request(&node->active, rq); | ||
| 25 | } | ||
| 26 | |||
| 27 | static inline void | ||
| 28 | intel_gt_buffer_pool_put(struct intel_gt_buffer_pool_node *node) | ||
| 29 | { | ||
| 30 | i915_active_release(&node->active); | ||
| 31 | } | ||
| 32 | |||
| 33 | void intel_gt_init_buffer_pool(struct intel_gt *gt); | ||
| 34 | void intel_gt_flush_buffer_pool(struct intel_gt *gt); | ||
| 35 | void intel_gt_fini_buffer_pool(struct intel_gt *gt); | ||
| 36 | |||
| 37 | #endif /* INTEL_GT_BUFFER_POOL_H */ | ||
| @@ -4,26 +4,29 @@ · 3 unmodified lines | |||
| 4 | * Copyright © 2014-2018 Intel Corporation | 4 | * Copyright © 2014-2018 Intel Corporation |
| 5 | */ | 5 | */ |
| 6 | 6 | ||
| 7 | #ifndef INTEL_ENGINE_POOL_TYPES_H | 7 | #ifndef INTEL_GT_BUFFER_POOL_TYPES_H |
| 8 | #define INTEL_ENGINE_POOL_TYPES_H | 8 | #define INTEL_GT_BUFFER_POOL_TYPES_H |
| 9 | 9 | ||
| 10 | #include <linux/list.h> | 10 | #include <linux/list.h> |
| 11 | #include <linux/spinlock.h> | 11 | #include <linux/spinlock.h> |
| 12 | #include <linux/workqueue.h> | ||
| 12 | 13 | ||
| 13 | #include "i915_active_types.h" | 14 | #include "i915_active_types.h" |
| 14 | 15 | ||
| 15 | struct drm_i915_gem_object; | 16 | struct drm_i915_gem_object; |
| 16 | 17 | ||
| 17 | struct intel_engine_pool { | 18 | struct intel_gt_buffer_pool { |
| 18 | spinlock_t lock; | 19 | spinlock_t lock; |
| 19 | struct list_head cache_list[4]; | 20 | struct list_head cache_list[4]; |
| 21 | struct delayed_work work; | ||
| 20 | }; | 22 | }; |
| 21 | 23 | ||
| 22 | struct intel_engine_pool_node { | 24 | struct intel_gt_buffer_pool_node { |
| 23 | struct i915_active active; | 25 | struct i915_active active; |
| 24 | struct drm_i915_gem_object *obj; | 26 | struct drm_i915_gem_object *obj; |
| 25 | struct list_head link; | 27 | struct list_head link; |
| 26 | struct intel_engine_pool *pool; | 28 | struct intel_gt_buffer_pool *pool; |
| 29 | unsigned long age; | ||
| 27 | }; | 30 | }; |
| 28 | 31 | ||
| 29 | #endif /* INTEL_ENGINE_POOL_TYPES_H */ | 32 | #endif /* INTEL_GT_BUFFER_POOL_TYPES_H */ |
| @@ -150,10 +150,6 @@ static void gt_sanitize(struct intel_gt *gt, bool force) · 149 unmodified lines | |||
| 150 | if (intel_gt_is_wedged(gt)) | 150 | if (intel_gt_is_wedged(gt)) |
| 151 | intel_gt_unset_wedged(gt); | 151 | intel_gt_unset_wedged(gt); |
| 152 | 152 | ||
| 153 | for_each_engine(engine, gt, id) | ||
| 154 | if (engine->sanitize) | ||
| 155 | engine->sanitize(engine); | ||
| 156 | |||
| 157 | intel_uc_sanitize(>->uc); | 153 | intel_uc_sanitize(>->uc); |
| 158 | 154 | ||
| 159 | for_each_engine(engine, gt, id) | 155 | for_each_engine(engine, gt, id) |
| @@ -162,6 +158,10 @@ static void gt_sanitize(struct intel_gt *gt, bool force) · 2 unmodified lines | |||
| 162 | 158 | ||
| 163 | intel_uc_reset_prepare(>->uc); | 159 | intel_uc_reset_prepare(>->uc); |
| 164 | 160 | ||
| 161 | for_each_engine(engine, gt, id) | ||
| 162 | if (engine->sanitize) | ||
| 163 | engine->sanitize(engine); | ||
| 164 | |||
| 165 | if (reset_engines(gt) || force) { | 165 | if (reset_engines(gt) || force) { |
| 166 | for_each_engine(engine, gt, id) | 166 | for_each_engine(engine, gt, id) |
| 167 | __intel_engine_reset(engine, false); | 167 | __intel_engine_reset(engine, false); |
| @@ -171,6 +171,8 @@ static void gt_sanitize(struct intel_gt *gt, bool force) · 3 unmodified lines | |||
| 171 | if (engine->reset.finish) | 171 | if (engine->reset.finish) |
| 172 | engine->reset.finish(engine); | 172 | engine->reset.finish(engine); |
| 173 | 173 | ||
| 174 | intel_rps_sanitize(>->rps); | ||
| 175 | |||
| 174 | intel_uncore_forcewake_put(gt->uncore, FORCEWAKE_ALL); | 176 | intel_uncore_forcewake_put(gt->uncore, FORCEWAKE_ALL); |
| 175 | intel_runtime_pm_put(gt->uncore->rpm, wakeref); | 177 | intel_runtime_pm_put(gt->uncore->rpm, wakeref); |
| 176 | } | 178 | } |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | |||
| 17 | 17 | ||
| 18 | #include "i915_vma.h" | 18 | #include "i915_vma.h" |
| 19 | #include "intel_engine_types.h" | 19 | #include "intel_engine_types.h" |
| 20 | #include "intel_gt_buffer_pool_types.h" | ||
| 20 | #include "intel_llc_types.h" | 21 | #include "intel_llc_types.h" |
| 21 | #include "intel_reset_types.h" | 22 | #include "intel_reset_types.h" |
| 22 | #include "intel_rc6_types.h" | 23 | #include "intel_rc6_types.h" |
| @@ -97,6 +98,16 @@ struct intel_gt { · 74 unmodified lines | |||
| 97 | */ | 98 | */ |
| 98 | struct i915_address_space *vm; | 99 | struct i915_address_space *vm; |
| 99 | 100 | ||
| 101 | /* | ||
| 102 | * A pool of objects to use as shadow copies of client batch buffers | ||
| 103 | * when the command parser is enabled. Prevents the client from | ||
| 104 | * modifying the batch contents after software parsing. | ||
| 105 | * | ||
| 106 | * Buffers older than 1s are periodically reaped from the pool, | ||
| 107 | * or may be reclaimed by the shrinker before then. | ||
| 108 | */ | ||
| 109 | struct intel_gt_buffer_pool buffer_pool; | ||
| 110 | |||
| 100 | struct i915_vma *scratch; | 111 | struct i915_vma *scratch; |
| 101 | }; | 112 | }; |
| 102 | 113 | ||
| @@ -217,7 +217,7 @@ struct virtual_engine { · 216 unmodified lines | |||
| 217 | 217 | ||
| 218 | /* And finally, which physical engines this virtual engine maps onto. */ | 218 | /* And finally, which physical engines this virtual engine maps onto. */ |
| 219 | unsigned int num_siblings; | 219 | unsigned int num_siblings; |
| 220 | struct intel_engine_cs *siblings[0]; | 220 | struct intel_engine_cs *siblings[]; |
| 221 | }; | 221 | }; |
| 222 | 222 | ||
| 223 | static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine) | 223 | static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine) |
| @@ -429,18 +429,7 @@ static int effective_prio(const struct i915_request *rq) · 205 unmodified lines | |||
| 429 | if (i915_request_has_nopreempt(rq)) | 429 | if (i915_request_has_nopreempt(rq)) |
| 430 | prio = I915_PRIORITY_UNPREEMPTABLE; | 430 | prio = I915_PRIORITY_UNPREEMPTABLE; |
| 431 | 431 | ||
| 432 | /* | 432 | return prio; |
| 433 | * On unwinding the active request, we give it a priority bump | ||
| 434 | * if it has completed waiting on any semaphore. If we know that | ||
| 435 | * the request has already started, we can prevent an unwanted | ||
| 436 | * preempt-to-idle cycle by taking that into account now. | ||
| 437 | */ | ||
| 438 | if (__i915_request_has_started(rq)) | ||
| 439 | prio |= I915_PRIORITY_NOSEMAPHORE; | ||
| 440 | |||
| 441 | /* Restrict mere WAIT boosts from triggering preemption */ | ||
| 442 | BUILD_BUG_ON(__NO_PREEMPTION & ~I915_PRIORITY_MASK); /* only internal */ | ||
| 443 | return prio | __NO_PREEMPTION; | ||
| 444 | } | 433 | } |
| 445 | 434 | ||
| 446 | static int queue_prio(const struct intel_engine_execlists *execlists) | 435 | static int queue_prio(const struct intel_engine_execlists *execlists) |
| @@ -1271,14 +1260,11 @@ execlists_check_context(const struct intel_context *ce, · 824 unmodified lines | |||
| 1271 | static void restore_default_state(struct intel_context *ce, | 1260 | static void restore_default_state(struct intel_context *ce, |
| 1272 | struct intel_engine_cs *engine) | 1261 | struct intel_engine_cs *engine) |
| 1273 | { | 1262 | { |
| 1274 | u32 *regs = ce->lrc_reg_state; | 1263 | u32 *regs; |
| 1275 | 1264 | ||
| 1276 | if (engine->pinned_default_state) | 1265 | regs = memset(ce->lrc_reg_state, 0, engine->context_size - PAGE_SIZE); |
| 1277 | memcpy(regs, /* skip restoring the vanilla PPHWSP */ | 1266 | execlists_init_reg_state(regs, ce, engine, ce->ring, true); |
| 1278 | engine->pinned_default_state + LRC_STATE_OFFSET, | ||
| 1279 | engine->context_size - PAGE_SIZE); | ||
| 1280 | 1267 | ||
| 1281 | execlists_init_reg_state(regs, ce, engine, ce->ring, false); | ||
| 1282 | ce->runtime.last = intel_context_get_runtime(ce); | 1268 | ce->runtime.last = intel_context_get_runtime(ce); |
| 1283 | } | 1269 | } |
| 1284 | 1270 | ||
| @@ -1372,7 +1358,7 @@ __execlists_schedule_in(struct i915_request *rq) · 87 unmodified lines | |||
| 1372 | ce->lrc.ccid = ce->tag; | 1358 | ce->lrc.ccid = ce->tag; |
| 1373 | } else { | 1359 | } else { |
| 1374 | /* We don't need a strict matching tag, just different values */ | 1360 | /* We don't need a strict matching tag, just different values */ |
| 1375 | unsigned int tag = ffs(engine->context_tag); | 1361 | unsigned int tag = ffs(READ_ONCE(engine->context_tag)); |
| 1376 | 1362 | ||
| 1377 | GEM_BUG_ON(tag == 0 || tag >= BITS_PER_LONG); | 1363 | GEM_BUG_ON(tag == 0 || tag >= BITS_PER_LONG); |
| 1378 | clear_bit(tag - 1, &engine->context_tag); | 1364 | clear_bit(tag - 1, &engine->context_tag); |
| @@ -1826,30 +1812,16 @@ static bool virtual_matches(const struct virtual_engine *ve, · 447 unmodified lines | |||
| 1826 | return true; | 1812 | return true; |
| 1827 | } | 1813 | } |
| 1828 | 1814 | ||
| 1829 | static void virtual_xfer_breadcrumbs(struct virtual_engine *ve, | 1815 | static void virtual_xfer_breadcrumbs(struct virtual_engine *ve) |
| 1830 | struct i915_request *rq) | ||
| 1831 | { | 1816 | { |
| 1832 | struct intel_engine_cs *old = ve->siblings[0]; | 1817 | /* |
| 1833 | 1818 | * All the outstanding signals on ve->siblings[0] must have | |
| 1834 | /* All unattached (rq->engine == old) must already be completed */ | 1819 | * been completed, just pending the interrupt handler. As those |
| 1835 | 1820 | * signals still refer to the old sibling (via rq->engine), we must | |
| 1836 | spin_lock(&old->breadcrumbs.irq_lock); | 1821 | * transfer those to the old irq_worker to keep our locking |
| 1837 | if (!list_empty(&ve->context.signal_link)) { | 1822 | * consistent. |
| 1838 | list_del_init(&ve->context.signal_link); | 1823 | */ |
| 1839 | 1824 | intel_engine_transfer_stale_breadcrumbs(ve->siblings[0], &ve->context); | |
| 1840 | /* | ||
| 1841 | * We cannot acquire the new engine->breadcrumbs.irq_lock | ||
| 1842 | * (as we are holding a breadcrumbs.irq_lock already), | ||
| 1843 | * so attach this request to the signaler on submission. | ||
| 1844 | * The queued irq_work will occur when we finally drop | ||
| 1845 | * the engine->active.lock after dequeue. | ||
| 1846 | */ | ||
| 1847 | set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &rq->fence.flags); | ||
| 1848 | |||
| 1849 | /* Also transfer the pending irq_work for the old breadcrumb. */ | ||
| 1850 | intel_engine_signal_breadcrumbs(rq->engine); | ||
| 1851 | } | ||
| 1852 | spin_unlock(&old->breadcrumbs.irq_lock); | ||
| 1853 | } | 1825 | } |
| 1854 | 1826 | ||
| 1855 | #define for_each_waiter(p__, rq__) \ | 1827 | #define for_each_waiter(p__, rq__) \ |
| @@ -1883,12 +1855,16 @@ static void defer_request(struct i915_request *rq, struct list_head * const pl) · 27 unmodified lines | |||
| 1883 | struct i915_request *w = | 1855 | struct i915_request *w = |
| 1884 | container_of(p->waiter, typeof(*w), sched); | 1856 | container_of(p->waiter, typeof(*w), sched); |
| 1885 | 1857 | ||
| 1858 | if (p->flags & I915_DEPENDENCY_WEAK) | ||
| 1859 | continue; | ||
| 1860 | |||
| 1886 | /* Leave semaphores spinning on the other engines */ | 1861 | /* Leave semaphores spinning on the other engines */ |
| 1887 | if (w->engine != rq->engine) | 1862 | if (w->engine != rq->engine) |
| 1888 | continue; | 1863 | continue; |
| 1889 | 1864 | ||
| 1890 | /* No waiter should start before its signaler */ | 1865 | /* No waiter should start before its signaler */ |
| 1891 | GEM_BUG_ON(i915_request_started(w) && | 1866 | GEM_BUG_ON(i915_request_has_initial_breadcrumb(w) && |
| 1867 | i915_request_started(w) && | ||
| 1892 | !i915_request_completed(rq)); | 1868 | !i915_request_completed(rq)); |
| 1893 | 1869 | ||
| 1894 | GEM_BUG_ON(i915_request_is_active(w)); | 1870 | GEM_BUG_ON(i915_request_is_active(w)); |
| @@ -2280,7 +2256,7 @@ static void execlists_dequeue(struct intel_engine_cs *engine) · 385 unmodified lines | |||
| 2280 | engine); | 2256 | engine); |
| 2281 | 2257 | ||
| 2282 | if (!list_empty(&ve->context.signals)) | 2258 | if (!list_empty(&ve->context.signals)) |
| 2283 | virtual_xfer_breadcrumbs(ve, rq); | 2259 | virtual_xfer_breadcrumbs(ve); |
| 2284 | 2260 | ||
| 2285 | /* | 2261 | /* |
| 2286 | * Move the bound engine to the top of the list | 2262 | * Move the bound engine to the top of the list |
| @@ -3494,6 +3470,7 @@ static int gen8_emit_init_breadcrumb(struct i915_request *rq) · 1207 unmodified lines | |||
| 3494 | { | 3470 | { |
| 3495 | u32 *cs; | 3471 | u32 *cs; |
| 3496 | 3472 | ||
| 3473 | GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq)); | ||
| 3497 | if (!i915_request_timeline(rq)->has_initial_breadcrumb) | 3474 | if (!i915_request_timeline(rq)->has_initial_breadcrumb) |
| 3498 | return 0; | 3475 | return 0; |
| 3499 | 3476 | ||
| @@ -3520,6 +3497,56 @@ static int gen8_emit_init_breadcrumb(struct i915_request *rq) · 20 unmodified lines | |||
| 3520 | /* Record the updated position of the request's payload */ | 3497 | /* Record the updated position of the request's payload */ |
| 3521 | rq->infix = intel_ring_offset(rq, cs); | 3498 | rq->infix = intel_ring_offset(rq, cs); |
| 3522 | 3499 | ||
| 3500 | __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags); | ||
| 3501 | |||
| 3502 | return 0; | ||
| 3503 | } | ||
| 3504 | |||
| 3505 | static int emit_pdps(struct i915_request *rq) | ||
| 3506 | { | ||
| 3507 | const struct intel_engine_cs * const engine = rq->engine; | ||
| 3508 | struct i915_ppgtt * const ppgtt = i915_vm_to_ppgtt(rq->context->vm); | ||
| 3509 | int err, i; | ||
| 3510 | u32 *cs; | ||
| 3511 | |||
| 3512 | GEM_BUG_ON(intel_vgpu_active(rq->i915)); | ||
| 3513 | |||
| 3514 | /* | ||
| 3515 | * Beware ye of the dragons, this sequence is magic! | ||
| 3516 | * | ||
| 3517 | * Small changes to this sequence can cause anything from | ||
| 3518 | * GPU hangs to forcewake errors and machine lockups! | ||
| 3519 | */ | ||
| 3520 | |||
| 3521 | /* Flush any residual operations from the context load */ | ||
| 3522 | err = engine->emit_flush(rq, EMIT_FLUSH); | ||
| 3523 | if (err) | ||
| 3524 | return err; | ||
| 3525 | |||
| 3526 | /* Magic required to prevent forcewake errors! */ | ||
| 3527 | err = engine->emit_flush(rq, EMIT_INVALIDATE); | ||
| 3528 | if (err) | ||
| 3529 | return err; | ||
| 3530 | |||
| 3531 | cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2); | ||
| 3532 | if (IS_ERR(cs)) | ||
| 3533 | return PTR_ERR(cs); | ||
| 3534 | |||
| 3535 | /* Ensure the LRI have landed before we invalidate & continue */ | ||
| 3536 | *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED; | ||
| 3537 | for (i = GEN8_3LVL_PDPES; i--; ) { | ||
| 3538 | const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i); | ||
| 3539 | u32 base = engine->mmio_base; | ||
| 3540 | |||
| 3541 | *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i)); | ||
| 3542 | *cs++ = upper_32_bits(pd_daddr); | ||
| 3543 | *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i)); | ||
| 3544 | *cs++ = lower_32_bits(pd_daddr); | ||
| 3545 | } | ||
| 3546 | *cs++ = MI_NOOP; | ||
| 3547 | |||
| 3548 | intel_ring_advance(rq, cs); | ||
| 3549 | |||
| 3523 | return 0; | 3550 | return 0; |
| 3524 | } | 3551 | } |
| 3525 | 3552 | ||
| @@ -3544,6 +3571,12 @@ static int execlists_request_alloc(struct i915_request *request) · 18 unmodified lines | |||
| 3544 | * to cancel/unwind this request now. | 3571 | * to cancel/unwind this request now. |
| 3545 | */ | 3572 | */ |
| 3546 | 3573 | ||
| 3574 | if (!i915_vm_is_4lvl(request->context->vm)) { | ||
| 3575 | ret = emit_pdps(request); | ||
| 3576 | if (ret) | ||
| 3577 | return ret; | ||
| 3578 | } | ||
| 3579 | |||
| 3547 | /* Unconditionally invalidate GPU caches and TLBs. */ | 3580 | /* Unconditionally invalidate GPU caches and TLBs. */ |
| 3548 | ret = request->engine->emit_flush(request, EMIT_INVALIDATE); | 3581 | ret = request->engine->emit_flush(request, EMIT_INVALIDATE); |
| 3549 | if (ret) | 3582 | if (ret) |
| @@ -3885,6 +3918,14 @@ static void reset_csb_pointers(struct intel_engine_cs *engine) · 335 unmodified lines | |||
| 3885 | 3918 | ||
| 3886 | ring_set_paused(engine, 0); | 3919 | ring_set_paused(engine, 0); |
| 3887 | 3920 | ||
| 3921 | /* | ||
| 3922 | * Sometimes Icelake forgets to reset its pointers on a GPU reset. | ||
| 3923 | * Bludgeon them with a mmio update to be sure. | ||
| 3924 | */ | ||
| 3925 | ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR, | ||
| 3926 | 0xffff << 16 | reset_value << 8 | reset_value); | ||
| 3927 | ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR); | ||
| 3928 | |||
| 3888 | /* | 3929 | /* |
| 3889 | * After a reset, the HW starts writing into CSB entry [0]. We | 3930 | * After a reset, the HW starts writing into CSB entry [0]. We |
| 3890 | * therefore have to set our HEAD pointer back one entry so that | 3931 | * therefore have to set our HEAD pointer back one entry so that |
| @@ -3898,16 +3939,15 @@ static void reset_csb_pointers(struct intel_engine_cs *engine) · 7 unmodified lines | |||
| 3898 | WRITE_ONCE(*execlists->csb_write, reset_value); | 3939 | WRITE_ONCE(*execlists->csb_write, reset_value); |
| 3899 | wmb(); /* Make sure this is visible to HW (paranoia?) */ | 3940 | wmb(); /* Make sure this is visible to HW (paranoia?) */ |
| 3900 | 3941 | ||
| 3901 | /* | 3942 | invalidate_csb_entries(&execlists->csb_status[0], |
| 3902 | * Sometimes Icelake forgets to reset its pointers on a GPU reset. | 3943 | &execlists->csb_status[reset_value]); |
| 3903 | * Bludgeon them with a mmio update to be sure. | 3944 | |
| 3904 | */ | 3945 | /* Once more for luck and our trusty paranoia */ |
| 3905 | ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR, | 3946 | ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR, |
| 3906 | reset_value << 8 | reset_value); | 3947 | 0xffff << 16 | reset_value << 8 | reset_value); |
| 3907 | ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR); | 3948 | ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR); |
| 3908 | 3949 | ||
| 3909 | invalidate_csb_entries(&execlists->csb_status[0], | 3950 | GEM_BUG_ON(READ_ONCE(*execlists->csb_write) != reset_value); |
| 3910 | &execlists->csb_status[reset_value]); | ||
| 3911 | } | 3951 | } |
| 3912 | 3952 | ||
| 3913 | static void execlists_sanitize(struct intel_engine_cs *engine) | 3953 | static void execlists_sanitize(struct intel_engine_cs *engine) |
| @@ -4074,6 +4114,8 @@ static void execlists_reset_prepare(struct intel_engine_cs *engine) · 160 unmodified lines | |||
| 4074 | */ | 4114 | */ |
| 4075 | ring_set_paused(engine, 1); | 4115 | ring_set_paused(engine, 1); |
| 4076 | intel_engine_stop_cs(engine); | 4116 | intel_engine_stop_cs(engine); |
| 4117 | |||
| 4118 | engine->execlists.reset_ccid = active_ccid(engine); | ||
| 4077 | } | 4119 | } |
| 4078 | 4120 | ||
| 4079 | static void __reset_stop_ring(u32 *regs, const struct intel_engine_cs *engine) | 4121 | static void __reset_stop_ring(u32 *regs, const struct intel_engine_cs *engine) |
| @@ -4116,7 +4158,7 @@ static void __execlists_reset(struct intel_engine_cs *engine, bool stalled) · 36 unmodified lines | |||
| 4116 | * its request, it was still running at the time of the | 4158 | * its request, it was still running at the time of the |
| 4117 | * reset and will have been clobbered. | 4159 | * reset and will have been clobbered. |
| 4118 | */ | 4160 | */ |
| 4119 | rq = execlists_active(execlists); | 4161 | rq = active_context(engine, engine->execlists.reset_ccid); |
| 4120 | if (!rq) | 4162 | if (!rq) |
| 4121 | goto unwind; | 4163 | goto unwind; |
| 4122 | 4164 | ||
| @@ -4166,8 +4208,6 @@ static void __execlists_reset(struct intel_engine_cs *engine, bool stalled) · 43 unmodified lines | |||
| 4166 | * image back to the expected values to skip over the guilty request. | 4208 | * image back to the expected values to skip over the guilty request. |
| 4167 | */ | 4209 | */ |
| 4168 | __i915_request_reset(rq, stalled); | 4210 | __i915_request_reset(rq, stalled); |
| 4169 | if (!stalled) | ||
| 4170 | goto out_replay; | ||
| 4171 | 4211 | ||
| 4172 | /* | 4212 | /* |
| 4173 | * We want a simple context + ring to execute the breadcrumb update. | 4213 | * We want a simple context + ring to execute the breadcrumb update. |
| @@ -4177,9 +4217,6 @@ static void __execlists_reset(struct intel_engine_cs *engine, bool stalled) · 3 unmodified lines | |||
| 4177 | * future request will be after userspace has had the opportunity | 4217 | * future request will be after userspace has had the opportunity |
| 4178 | * to recreate its own state. | 4218 | * to recreate its own state. |
| 4179 | */ | 4219 | */ |
| 4180 | GEM_BUG_ON(!intel_context_is_pinned(ce)); | ||
| 4181 | restore_default_state(ce, engine); | ||
| 4182 | |||
| 4183 | out_replay: | 4220 | out_replay: |
| 4184 | ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n", | 4221 | ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n", |
| 4185 | head, ce->ring->tail); | 4222 | head, ce->ring->tail); |
| @@ -4545,6 +4582,42 @@ static u32 preparser_disable(bool state) · 359 unmodified lines | |||
| 4545 | return MI_ARB_CHECK | 1 << 8 | state; | 4582 | return MI_ARB_CHECK | 1 << 8 | state; |
| 4546 | } | 4583 | } |
| 4547 | 4584 | ||
| 4585 | static i915_reg_t aux_inv_reg(const struct intel_engine_cs *engine) | ||
| 4586 | { | ||
| 4587 | static const i915_reg_t vd[] = { | ||
| 4588 | GEN12_VD0_AUX_NV, | ||
| 4589 | GEN12_VD1_AUX_NV, | ||
| 4590 | GEN12_VD2_AUX_NV, | ||
| 4591 | GEN12_VD3_AUX_NV, | ||
| 4592 | }; | ||
| 4593 | |||
| 4594 | static const i915_reg_t ve[] = { | ||
| 4595 | GEN12_VE0_AUX_NV, | ||
| 4596 | GEN12_VE1_AUX_NV, | ||
| 4597 | }; | ||
| 4598 | |||
| 4599 | if (engine->class == VIDEO_DECODE_CLASS) | ||
| 4600 | return vd[engine->instance]; | ||
| 4601 | |||
| 4602 | if (engine->class == VIDEO_ENHANCEMENT_CLASS) | ||
| 4603 | return ve[engine->instance]; | ||
| 4604 | |||
| 4605 | GEM_BUG_ON("unknown aux_inv_reg\n"); | ||
| 4606 | |||
| 4607 | return INVALID_MMIO_REG; | ||
| 4608 | } | ||
| 4609 | |||
| 4610 | static u32 * | ||
| 4611 | gen12_emit_aux_table_inv(const i915_reg_t inv_reg, u32 *cs) | ||
| 4612 | { | ||
| 4613 | *cs++ = MI_LOAD_REGISTER_IMM(1); | ||
| 4614 | *cs++ = i915_mmio_reg_offset(inv_reg); | ||
| 4615 | *cs++ = AUX_INV; | ||
| 4616 | *cs++ = MI_NOOP; | ||
| 4617 | |||
| 4618 | return cs; | ||
| 4619 | } | ||
| 4620 | |||
| 4548 | static int gen12_emit_flush_render(struct i915_request *request, | 4621 | static int gen12_emit_flush_render(struct i915_request *request, |
| 4549 | u32 mode) | 4622 | u32 mode) |
| 4550 | { | 4623 | { |
| @@ -4553,13 +4626,13 @@ static int gen12_emit_flush_render(struct i915_request *request, · 2 unmodified lines | |||
| 4553 | u32 *cs; | 4626 | u32 *cs; |
| 4554 | 4627 | ||
| 4555 | flags |= PIPE_CONTROL_TILE_CACHE_FLUSH; | 4628 | flags |= PIPE_CONTROL_TILE_CACHE_FLUSH; |
| 4629 | flags |= PIPE_CONTROL_FLUSH_L3; | ||
| 4556 | flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH; | 4630 | flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH; |
| 4557 | flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH; | 4631 | flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH; |
| 4558 | /* Wa_1409600907:tgl */ | 4632 | /* Wa_1409600907:tgl */ |
| 4559 | flags |= PIPE_CONTROL_DEPTH_STALL; | 4633 | flags |= PIPE_CONTROL_DEPTH_STALL; |
| 4560 | flags |= PIPE_CONTROL_DC_FLUSH_ENABLE; | 4634 | flags |= PIPE_CONTROL_DC_FLUSH_ENABLE; |
| 4561 | flags |= PIPE_CONTROL_FLUSH_ENABLE; | 4635 | flags |= PIPE_CONTROL_FLUSH_ENABLE; |
| 4562 | flags |= PIPE_CONTROL_HDC_PIPELINE_FLUSH; | ||
| 4563 | 4636 | ||
| 4564 | flags |= PIPE_CONTROL_STORE_DATA_INDEX; | 4637 | flags |= PIPE_CONTROL_STORE_DATA_INDEX; |
| 4565 | flags |= PIPE_CONTROL_QW_WRITE; | 4638 | flags |= PIPE_CONTROL_QW_WRITE; |
| @@ -4570,7 +4643,9 @@ static int gen12_emit_flush_render(struct i915_request *request, · 4 unmodified lines | |||
| 4570 | if (IS_ERR(cs)) | 4643 | if (IS_ERR(cs)) |
| 4571 | return PTR_ERR(cs); | 4644 | return PTR_ERR(cs); |
| 4572 | 4645 | ||
| 4573 | cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR); | 4646 | cs = gen12_emit_pipe_control(cs, |
| 4647 | PIPE_CONTROL0_HDC_PIPELINE_FLUSH, | ||
| 4648 | flags, LRC_PPHWSP_SCRATCH_ADDR); | ||
| 4574 | intel_ring_advance(request, cs); | 4649 | intel_ring_advance(request, cs); |
| 4575 | } | 4650 | } |
| 4576 | 4651 | ||
| @@ -4585,14 +4660,13 @@ static int gen12_emit_flush_render(struct i915_request *request, · 8 unmodified lines | |||
| 4585 | flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE; | 4660 | flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE; |
| 4586 | flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE; | 4661 | flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE; |
| 4587 | flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE; | 4662 | flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE; |
| 4588 | flags |= PIPE_CONTROL_L3_RO_CACHE_INVALIDATE; | ||
| 4589 | 4663 | ||
| 4590 | flags |= PIPE_CONTROL_STORE_DATA_INDEX; | 4664 | flags |= PIPE_CONTROL_STORE_DATA_INDEX; |
| 4591 | flags |= PIPE_CONTROL_QW_WRITE; | 4665 | flags |= PIPE_CONTROL_QW_WRITE; |
| 4592 | 4666 | ||
| 4593 | flags |= PIPE_CONTROL_CS_STALL; | 4667 | flags |= PIPE_CONTROL_CS_STALL; |
| 4594 | 4668 | ||
| 4595 | cs = intel_ring_begin(request, 8); | 4669 | cs = intel_ring_begin(request, 8 + 4); |
| 4596 | if (IS_ERR(cs)) | 4670 | if (IS_ERR(cs)) |
| 4597 | return PTR_ERR(cs); | 4671 | return PTR_ERR(cs); |
| 4598 | 4672 | ||
| @@ -4605,6 +4679,9 @@ static int gen12_emit_flush_render(struct i915_request *request, · 6 unmodified lines | |||
| 4605 | 4679 | ||
| 4606 | cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR); | 4680 | cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR); |
| 4607 | 4681 | ||
| 4682 | /* hsdes: 1809175790 */ | ||
| 4683 | cs = gen12_emit_aux_table_inv(GEN12_GFX_CCS_AUX_NV, cs); | ||
| 4684 | |||
| 4608 | *cs++ = preparser_disable(false); | 4685 | *cs++ = preparser_disable(false); |
| 4609 | intel_ring_advance(request, cs); | 4686 | intel_ring_advance(request, cs); |
| 4610 | } | 4687 | } |
| @@ -4612,6 +4689,56 @@ static int gen12_emit_flush_render(struct i915_request *request, · 1 unmodified line | |||
| 4612 | return 0; | 4689 | return 0; |
| 4613 | } | 4690 | } |
| 4614 | 4691 | ||
| 4692 | static int gen12_emit_flush(struct i915_request *request, u32 mode) | ||
| 4693 | { | ||
| 4694 | intel_engine_mask_t aux_inv = 0; | ||
| 4695 | u32 cmd, *cs; | ||
| 4696 | |||
| 4697 | if (mode & EMIT_INVALIDATE) | ||
| 4698 | aux_inv = request->engine->mask & ~BIT(BCS0); | ||
| 4699 | |||
| 4700 | cs = intel_ring_begin(request, | ||
| 4701 | 4 + (aux_inv ? 2 * hweight8(aux_inv) + 2 : 0)); | ||
| 4702 | if (IS_ERR(cs)) | ||
| 4703 | return PTR_ERR(cs); | ||
| 4704 | |||
| 4705 | cmd = MI_FLUSH_DW + 1; | ||
| 4706 | |||
| 4707 | /* We always require a command barrier so that subsequent | ||
| 4708 | * commands, such as breadcrumb interrupts, are strictly ordered | ||
| 4709 | * wrt the contents of the write cache being flushed to memory | ||
| 4710 | * (and thus being coherent from the CPU). | ||
| 4711 | */ | ||
| 4712 | cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW; | ||
| 4713 | |||
| 4714 | if (mode & EMIT_INVALIDATE) { | ||
| 4715 | cmd |= MI_INVALIDATE_TLB; | ||
| 4716 | if (request->engine->class == VIDEO_DECODE_CLASS) | ||
| 4717 | cmd |= MI_INVALIDATE_BSD; | ||
| 4718 | } | ||
| 4719 | |||
| 4720 | *cs++ = cmd; | ||
| 4721 | *cs++ = LRC_PPHWSP_SCRATCH_ADDR; | ||
| 4722 | *cs++ = 0; /* upper addr */ | ||
| 4723 | *cs++ = 0; /* value */ | ||
| 4724 | |||
| 4725 | if (aux_inv) { /* hsdes: 1809175790 */ | ||
| 4726 | struct intel_engine_cs *engine; | ||
| 4727 | unsigned int tmp; | ||
| 4728 | |||
| 4729 | *cs++ = MI_LOAD_REGISTER_IMM(hweight8(aux_inv)); | ||
| 4730 | for_each_engine_masked(engine, request->engine->gt, | ||
| 4731 | aux_inv, tmp) { | ||
| 4732 | *cs++ = i915_mmio_reg_offset(aux_inv_reg(engine)); | ||
| 4733 | *cs++ = AUX_INV; | ||
| 4734 | } | ||
| 4735 | *cs++ = MI_NOOP; | ||
| 4736 | } | ||
| 4737 | intel_ring_advance(request, cs); | ||
| 4738 | |||
| 4739 | return 0; | ||
| 4740 | } | ||
| 4741 | |||
| 4615 | /* | 4742 | /* |
| 4616 | * Reserve space for 2 NOOPs at the end of each request to be | 4743 | * Reserve space for 2 NOOPs at the end of each request to be |
| 4617 | * used as a workaround for not being allowed to do lite | 4744 | * used as a workaround for not being allowed to do lite |
| @@ -4641,8 +4768,7 @@ static u32 *emit_preempt_busywait(struct i915_request *request, u32 *cs) · 23 unmodified lines | |||
| 4641 | } | 4768 | } |
| 4642 | 4769 | ||
| 4643 | static __always_inline u32* | 4770 | static __always_inline u32* |
| 4644 | gen8_emit_fini_breadcrumb_footer(struct i915_request *request, | 4771 | gen8_emit_fini_breadcrumb_tail(struct i915_request *request, u32 *cs) |
| 4645 | u32 *cs) | ||
| 4646 | { | 4772 | { |
| 4647 | *cs++ = MI_USER_INTERRUPT; | 4773 | *cs++ = MI_USER_INTERRUPT; |
| 4648 | 4774 | ||
| @@ -4656,14 +4782,16 @@ gen8_emit_fini_breadcrumb_footer(struct i915_request *request, · 7 unmodified lines | |||
| 4656 | return gen8_emit_wa_tail(request, cs); | 4782 | return gen8_emit_wa_tail(request, cs); |
| 4657 | } | 4783 | } |
| 4658 | 4784 | ||
| 4659 | static u32 *gen8_emit_fini_breadcrumb(struct i915_request *request, u32 *cs) | 4785 | static u32 *emit_xcs_breadcrumb(struct i915_request *request, u32 *cs) |
| 4660 | { | 4786 | { |
| 4661 | cs = gen8_emit_ggtt_write(cs, | 4787 | u32 addr = i915_request_active_timeline(request)->hwsp_offset; |
| 4662 | request->fence.seqno, | 4788 | |
| 4663 | i915_request_active_timeline(request)->hwsp_offset, | 4789 | return gen8_emit_ggtt_write(cs, request->fence.seqno, addr, 0); |
| 4664 | 0); | 4790 | } |
| 4665 | 4791 | ||
| 4666 | return gen8_emit_fini_breadcrumb_footer(request, cs); | 4792 | static u32 *gen8_emit_fini_breadcrumb(struct i915_request *rq, u32 *cs) |
| 4793 | { | ||
| 4794 | return gen8_emit_fini_breadcrumb_tail(rq, emit_xcs_breadcrumb(rq, cs)); | ||
| 4667 | } | 4795 | } |
| 4668 | 4796 | ||
| 4669 | static u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) | 4797 | static u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) |
| @@ -4681,7 +4809,7 @@ static u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) · 11 unmodified lines | |||
| 4681 | PIPE_CONTROL_FLUSH_ENABLE | | 4809 | PIPE_CONTROL_FLUSH_ENABLE | |
| 4682 | PIPE_CONTROL_CS_STALL); | 4810 | PIPE_CONTROL_CS_STALL); |
| 4683 | 4811 | ||
| 4684 | return gen8_emit_fini_breadcrumb_footer(request, cs); | 4812 | return gen8_emit_fini_breadcrumb_tail(request, cs); |
| 4685 | } | 4813 | } |
| 4686 | 4814 | ||
| 4687 | static u32 * | 4815 | static u32 * |
| @@ -4697,7 +4825,7 @@ gen11_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) · 9 unmodified lines | |||
| 4697 | PIPE_CONTROL_DC_FLUSH_ENABLE | | 4825 | PIPE_CONTROL_DC_FLUSH_ENABLE | |
| 4698 | PIPE_CONTROL_FLUSH_ENABLE); | 4826 | PIPE_CONTROL_FLUSH_ENABLE); |
| 4699 | 4827 | ||
| 4700 | return gen8_emit_fini_breadcrumb_footer(request, cs); | 4828 | return gen8_emit_fini_breadcrumb_tail(request, cs); |
| 4701 | } | 4829 | } |
| 4702 | 4830 | ||
| 4703 | /* | 4831 | /* |
| @@ -4735,7 +4863,7 @@ static u32 *gen12_emit_preempt_busywait(struct i915_request *request, u32 *cs) · 31 unmodified lines | |||
| 4735 | } | 4863 | } |
| 4736 | 4864 | ||
| 4737 | static __always_inline u32* | 4865 | static __always_inline u32* |
| 4738 | gen12_emit_fini_breadcrumb_footer(struct i915_request *request, u32 *cs) | 4866 | gen12_emit_fini_breadcrumb_tail(struct i915_request *request, u32 *cs) |
| 4739 | { | 4867 | { |
| 4740 | *cs++ = MI_USER_INTERRUPT; | 4868 | *cs++ = MI_USER_INTERRUPT; |
| 4741 | 4869 | ||
| @@ -4749,33 +4877,29 @@ gen12_emit_fini_breadcrumb_footer(struct i915_request *request, u32 *cs) · 7 unmodified lines | |||
| 4749 | return gen8_emit_wa_tail(request, cs); | 4877 | return gen8_emit_wa_tail(request, cs); |
| 4750 | } | 4878 | } |
| 4751 | 4879 | ||
| 4752 | static u32 *gen12_emit_fini_breadcrumb(struct i915_request *request, u32 *cs) | 4880 | static u32 *gen12_emit_fini_breadcrumb(struct i915_request *rq, u32 *cs) |
| 4753 | { | 4881 | { |
| 4754 | cs = gen8_emit_ggtt_write(cs, | 4882 | return gen12_emit_fini_breadcrumb_tail(rq, emit_xcs_breadcrumb(rq, cs)); |
| 4755 | request->fence.seqno, | ||
| 4756 | i915_request_active_timeline(request)->hwsp_offset, | ||
| 4757 | 0); | ||
| 4758 | |||
| 4759 | return gen12_emit_fini_breadcrumb_footer(request, cs); | ||
| 4760 | } | 4883 | } |
| 4761 | 4884 | ||
| 4762 | static u32 * | 4885 | static u32 * |
| 4763 | gen12_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) | 4886 | gen12_emit_fini_breadcrumb_rcs(struct i915_request *request, u32 *cs) |
| 4764 | { | 4887 | { |
| 4765 | cs = gen8_emit_ggtt_write_rcs(cs, | 4888 | cs = gen12_emit_ggtt_write_rcs(cs, |
| 4766 | request->fence.seqno, | 4889 | request->fence.seqno, |
| 4767 | i915_request_active_timeline(request)->hwsp_offset, | 4890 | i915_request_active_timeline(request)->hwsp_offset, |
| 4768 | PIPE_CONTROL_CS_STALL | | 4891 | PIPE_CONTROL0_HDC_PIPELINE_FLUSH, |
| 4769 | PIPE_CONTROL_TILE_CACHE_FLUSH | | 4892 | PIPE_CONTROL_CS_STALL | |
| 4770 | PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | | 4893 | PIPE_CONTROL_TILE_CACHE_FLUSH | |
| 4771 | PIPE_CONTROL_DEPTH_CACHE_FLUSH | | 4894 | PIPE_CONTROL_FLUSH_L3 | |
| 4772 | /* Wa_1409600907:tgl */ | 4895 | PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH | |
| 4773 | PIPE_CONTROL_DEPTH_STALL | | 4896 | PIPE_CONTROL_DEPTH_CACHE_FLUSH | |
| 4774 | PIPE_CONTROL_DC_FLUSH_ENABLE | | 4897 | /* Wa_1409600907:tgl */ |
| 4775 | PIPE_CONTROL_FLUSH_ENABLE | | 4898 | PIPE_CONTROL_DEPTH_STALL | |
| 4776 | PIPE_CONTROL_HDC_PIPELINE_FLUSH); | 4899 | PIPE_CONTROL_DC_FLUSH_ENABLE | |
| 4900 | PIPE_CONTROL_FLUSH_ENABLE); | ||
| 4777 | 4901 | ||
| 4778 | return gen12_emit_fini_breadcrumb_footer(request, cs); | 4902 | return gen12_emit_fini_breadcrumb_tail(request, cs); |
| 4779 | } | 4903 | } |
| 4780 | 4904 | ||
| 4781 | static void execlists_park(struct intel_engine_cs *engine) | 4905 | static void execlists_park(struct intel_engine_cs *engine) |
| @@ -4801,8 +4925,11 @@ void intel_execlists_set_default_submission(struct intel_engine_cs *engine) · 19 unmodified lines | |||
| 4801 | engine->flags |= I915_ENGINE_SUPPORTS_STATS; | 4925 | engine->flags |= I915_ENGINE_SUPPORTS_STATS; |
| 4802 | if (!intel_vgpu_active(engine->i915)) { | 4926 | if (!intel_vgpu_active(engine->i915)) { |
| 4803 | engine->flags |= I915_ENGINE_HAS_SEMAPHORES; | 4927 | engine->flags |= I915_ENGINE_HAS_SEMAPHORES; |
| 4804 | if (HAS_LOGICAL_RING_PREEMPTION(engine->i915)) | 4928 | if (HAS_LOGICAL_RING_PREEMPTION(engine->i915)) { |
| 4805 | engine->flags |= I915_ENGINE_HAS_PREEMPTION; | 4929 | engine->flags |= I915_ENGINE_HAS_PREEMPTION; |
| 4930 | if (IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION)) | ||
| 4931 | engine->flags |= I915_ENGINE_HAS_TIMESLICES; | ||
| 4932 | } | ||
| 4806 | } | 4933 | } |
| 4807 | 4934 | ||
| 4808 | if (INTEL_GEN(engine->i915) >= 12) | 4935 | if (INTEL_GEN(engine->i915) >= 12) |
| @@ -4845,9 +4972,10 @@ logical_ring_default_vfuncs(struct intel_engine_cs *engine) · 36 unmodified lines | |||
| 4845 | engine->emit_flush = gen8_emit_flush; | 4972 | engine->emit_flush = gen8_emit_flush; |
| 4846 | engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb; | 4973 | engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb; |
| 4847 | engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb; | 4974 | engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb; |
| 4848 | if (INTEL_GEN(engine->i915) >= 12) | 4975 | if (INTEL_GEN(engine->i915) >= 12) { |
| 4849 | engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb; | 4976 | engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb; |
| 4850 | 4977 | engine->emit_flush = gen12_emit_flush; | |
| 4978 | } | ||
| 4851 | engine->set_default_submission = intel_execlists_set_default_submission; | 4979 | engine->set_default_submission = intel_execlists_set_default_submission; |
| 4852 | 4980 | ||
| 4853 | if (INTEL_GEN(engine->i915) < 11) { | 4981 | if (INTEL_GEN(engine->i915) < 11) { |
| @@ -113,7 +113,6 @@ static void gen9_rc6_enable(struct intel_rc6 *rc6) · 112 unmodified lines | |||
| 113 | struct intel_uncore *uncore = rc6_to_uncore(rc6); | 113 | struct intel_uncore *uncore = rc6_to_uncore(rc6); |
| 114 | struct intel_engine_cs *engine; | 114 | struct intel_engine_cs *engine; |
| 115 | enum intel_engine_id id; | 115 | enum intel_engine_id id; |
| 116 | u32 rc6_mode; | ||
| 117 | 116 | ||
| 118 | /* 2b: Program RC6 thresholds.*/ | 117 | /* 2b: Program RC6 thresholds.*/ |
| 119 | if (INTEL_GEN(rc6_to_i915(rc6)) >= 10) { | 118 | if (INTEL_GEN(rc6_to_i915(rc6)) >= 10) { |
| @@ -165,16 +164,11 @@ static void gen9_rc6_enable(struct intel_rc6 *rc6) · 45 unmodified lines | |||
| 165 | /* 3a: Enable RC6 */ | 164 | /* 3a: Enable RC6 */ |
| 166 | set(uncore, GEN6_RC6_THRESHOLD, 37500); /* 37.5/125ms per EI */ | 165 | set(uncore, GEN6_RC6_THRESHOLD, 37500); /* 37.5/125ms per EI */ |
| 167 | 166 | ||
| 168 | /* WaRsUseTimeoutMode:cnl (pre-prod) */ | ||
| 169 | if (IS_CNL_REVID(rc6_to_i915(rc6), CNL_REVID_A0, CNL_REVID_C0)) | ||
| 170 | rc6_mode = GEN7_RC_CTL_TO_MODE; | ||
| 171 | else | ||
| 172 | rc6_mode = GEN6_RC_CTL_EI_MODE(1); | ||
| 173 | 167 | ||
| 174 | rc6->ctl_enable = | 168 | rc6->ctl_enable = |
| 175 | GEN6_RC_CTL_HW_ENABLE | | 169 | GEN6_RC_CTL_HW_ENABLE | |
| 176 | GEN6_RC_CTL_RC6_ENABLE | | 170 | GEN6_RC_CTL_RC6_ENABLE | |
| 177 | rc6_mode; | 171 | GEN6_RC_CTL_EI_MODE(1); |
| 178 | 172 | ||
| 179 | /* | 173 | /* |
| 180 | * WaRsDisableCoarsePowerGating:skl,cnl | 174 | * WaRsDisableCoarsePowerGating:skl,cnl |
| @@ -219,6 +219,14 @@ int intel_renderstate_emit(struct intel_renderstate *so, · 218 unmodified lines | |||
| 219 | if (!so->vma) | 219 | if (!so->vma) |
| 220 | return 0; | 220 | return 0; |
| 221 | 221 | ||
| 222 | i915_vma_lock(so->vma); | ||
| 223 | err = i915_request_await_object(rq, so->vma->obj, false); | ||
| 224 | if (err == 0) | ||
| 225 | err = i915_vma_move_to_active(so->vma, rq, 0); | ||
| 226 | i915_vma_unlock(so->vma); | ||
| 227 | if (err) | ||
| 228 | return err; | ||
| 229 | |||
| 222 | err = engine->emit_bb_start(rq, | 230 | err = engine->emit_bb_start(rq, |
| 223 | so->batch_offset, so->batch_size, | 231 | so->batch_offset, so->batch_size, |
| 224 | I915_DISPATCH_SECURE); | 232 | I915_DISPATCH_SECURE); |
| @@ -233,13 +241,7 @@ int intel_renderstate_emit(struct intel_renderstate *so, · 8 unmodified lines | |||
| 233 | return err; | 241 | return err; |
| 234 | } | 242 | } |
| 235 | 243 | ||
| 236 | i915_vma_lock(so->vma); | 244 | return 0; |
| 237 | err = i915_request_await_object(rq, so->vma->obj, false); | ||
| 238 | if (err == 0) | ||
| 239 | err = i915_vma_move_to_active(so->vma, rq, 0); | ||
| 240 | i915_vma_unlock(so->vma); | ||
| 241 | |||
| 242 | return err; | ||
| 243 | } | 245 | } |
| 244 | 246 | ||
| 245 | void intel_renderstate_fini(struct intel_renderstate *so) | 247 | void intel_renderstate_fini(struct intel_renderstate *so) |
| @@ -1844,8 +1844,11 @@ void intel_rps_init(struct intel_rps *rps) · 1843 unmodified lines | |||
| 1844 | 1844 | ||
| 1845 | if (INTEL_GEN(i915) >= 8 && INTEL_GEN(i915) < 11) | 1845 | if (INTEL_GEN(i915) >= 8 && INTEL_GEN(i915) < 11) |
| 1846 | rps->pm_intrmsk_mbz |= GEN8_PMINTR_DISABLE_REDIRECT_TO_GUC; | 1846 | rps->pm_intrmsk_mbz |= GEN8_PMINTR_DISABLE_REDIRECT_TO_GUC; |
| 1847 | } | ||
| 1847 | 1848 | ||
| 1848 | if (INTEL_GEN(i915) >= 6) | 1849 | void intel_rps_sanitize(struct intel_rps *rps) |
| 1850 | { | ||
| 1851 | if (INTEL_GEN(rps_to_i915(rps)) >= 6) | ||
| 1849 | rps_disable_interrupts(rps); | 1852 | rps_disable_interrupts(rps); |
| 1850 | } | 1853 | } |
| 1851 | 1854 | ||
| @@ -13,6 +13,7 @@ struct i915_request; · 12 unmodified lines | |||
| 13 | 13 | ||
| 14 | void intel_rps_init_early(struct intel_rps *rps); | 14 | void intel_rps_init_early(struct intel_rps *rps); |
| 15 | void intel_rps_init(struct intel_rps *rps); | 15 | void intel_rps_init(struct intel_rps *rps); |
| 16 | void intel_rps_sanitize(struct intel_rps *rps); | ||
| 16 | 17 | ||
| 17 | void intel_rps_driver_register(struct intel_rps *rps); | 18 | void intel_rps_driver_register(struct intel_rps *rps); |
| 18 | void intel_rps_driver_unregister(struct intel_rps *rps); | 19 | void intel_rps_driver_unregister(struct intel_rps *rps); |
| @@ -211,9 +211,9 @@ static void cacheline_free(struct intel_timeline_cacheline *cl) · 210 unmodified lines | |||
| 211 | i915_active_release(&cl->active); | 211 | i915_active_release(&cl->active); |
| 212 | } | 212 | } |
| 213 | 213 | ||
| 214 | int intel_timeline_init(struct intel_timeline *timeline, | 214 | static int intel_timeline_init(struct intel_timeline *timeline, |
| 215 | struct intel_gt *gt, | 215 | struct intel_gt *gt, |
| 216 | struct i915_vma *hwsp) | 216 | struct i915_vma *hwsp) |
| 217 | { | 217 | { |
| 218 | void *vaddr; | 218 | void *vaddr; |
| 219 | 219 | ||
| @@ -280,7 +280,7 @@ void intel_gt_init_timelines(struct intel_gt *gt) · 60 unmodified lines | |||
| 280 | INIT_LIST_HEAD(&timelines->hwsp_free_list); | 280 | INIT_LIST_HEAD(&timelines->hwsp_free_list); |
| 281 | } | 281 | } |
| 282 | 282 | ||
| 283 | void intel_timeline_fini(struct intel_timeline *timeline) | 283 | static void intel_timeline_fini(struct intel_timeline *timeline) |
| 284 | { | 284 | { |
| 285 | GEM_BUG_ON(atomic_read(&timeline->pin_count)); | 285 | GEM_BUG_ON(atomic_read(&timeline->pin_count)); |
| 286 | GEM_BUG_ON(!list_empty(&timeline->requests)); | 286 | GEM_BUG_ON(!list_empty(&timeline->requests)); |
| @@ -31,11 +31,6 @@ · 30 unmodified lines | |||
| 31 | #include "i915_syncmap.h" | 31 | #include "i915_syncmap.h" |
| 32 | #include "gt/intel_timeline_types.h" | 32 | #include "gt/intel_timeline_types.h" |
| 33 | 33 | ||
| 34 | int intel_timeline_init(struct intel_timeline *tl, | ||
| 35 | struct intel_gt *gt, | ||
| 36 | struct i915_vma *hwsp); | ||
| 37 | void intel_timeline_fini(struct intel_timeline *tl); | ||
| 38 | |||
| 39 | struct intel_timeline * | 34 | struct intel_timeline * |
| 40 | intel_timeline_create(struct intel_gt *gt, struct i915_vma *global_hwsp); | 35 | intel_timeline_create(struct intel_gt *gt, struct i915_vma *global_hwsp); |
| 41 | 36 | ||
| @@ -485,25 +485,14 @@ static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine, · 484 unmodified lines | |||
| 485 | static void cnl_ctx_workarounds_init(struct intel_engine_cs *engine, | 485 | static void cnl_ctx_workarounds_init(struct intel_engine_cs *engine, |
| 486 | struct i915_wa_list *wal) | 486 | struct i915_wa_list *wal) |
| 487 | { | 487 | { |
| 488 | struct drm_i915_private *i915 = engine->i915; | ||
| 489 | |||
| 490 | /* WaForceContextSaveRestoreNonCoherent:cnl */ | 488 | /* WaForceContextSaveRestoreNonCoherent:cnl */ |
| 491 | WA_SET_BIT_MASKED(CNL_HDC_CHICKEN0, | 489 | WA_SET_BIT_MASKED(CNL_HDC_CHICKEN0, |
| 492 | HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT); | 490 | HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT); |
| 493 | 491 | ||
| 494 | /* WaThrottleEUPerfToAvoidTDBackPressure:cnl(pre-prod) */ | ||
| 495 | if (IS_CNL_REVID(i915, CNL_REVID_B0, CNL_REVID_B0)) | ||
| 496 | WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, THROTTLE_12_5); | ||
| 497 | |||
| 498 | /* WaDisableReplayBufferBankArbitrationOptimization:cnl */ | 492 | /* WaDisableReplayBufferBankArbitrationOptimization:cnl */ |
| 499 | WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2, | 493 | WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2, |
| 500 | GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION); | 494 | GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION); |
| 501 | 495 | ||
| 502 | /* WaDisableEnhancedSBEVertexCaching:cnl (pre-prod) */ | ||
| 503 | if (IS_CNL_REVID(i915, 0, CNL_REVID_B0)) | ||
| 504 | WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2, | ||
| 505 | GEN8_CSC2_SBE_VUE_CACHE_CONSERVATIVE); | ||
| 506 | |||
| 507 | /* WaPushConstantDereferenceHoldDisable:cnl */ | 496 | /* WaPushConstantDereferenceHoldDisable:cnl */ |
| 508 | WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE); | 497 | WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE); |
| 509 | 498 | ||
| @@ -872,12 +861,6 @@ cnl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 362 unmodified lines | |||
| 872 | { | 861 | { |
| 873 | wa_init_mcr(i915, wal); | 862 | wa_init_mcr(i915, wal); |
| 874 | 863 | ||
| 875 | /* WaDisableI2mCycleOnWRPort:cnl (pre-prod) */ | ||
| 876 | if (IS_CNL_REVID(i915, CNL_REVID_B0, CNL_REVID_B0)) | ||
| 877 | wa_write_or(wal, | ||
| 878 | GAMT_CHKN_BIT_REG, | ||
| 879 | GAMT_CHKN_DISABLE_I2M_CYCLE_ON_WR_PORT); | ||
| 880 | |||
| 881 | /* WaInPlaceDecompressionHang:cnl */ | 864 | /* WaInPlaceDecompressionHang:cnl */ |
| 882 | wa_write_or(wal, | 865 | wa_write_or(wal, |
| 883 | GEN9_GAMT_ECO_REG_RW_IA, | 866 | GEN9_GAMT_ECO_REG_RW_IA, |
| @@ -934,10 +917,13 @@ icl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 50 unmodified lines | |||
| 934 | GAMT_CHKN_BIT_REG, | 917 | GAMT_CHKN_BIT_REG, |
| 935 | GAMT_CHKN_DISABLE_L3_COH_PIPE); | 918 | GAMT_CHKN_DISABLE_L3_COH_PIPE); |
| 936 | 919 | ||
| 937 | /* Wa_1607087056:icl */ | 920 | /* Wa_1607087056:icl,ehl,jsl */ |
| 938 | wa_write_or(wal, | 921 | if (IS_ICELAKE(i915) || |
| 939 | SLICE_UNIT_LEVEL_CLKGATE, | 922 | IS_EHL_REVID(i915, EHL_REVID_A0, EHL_REVID_A0)) { |
| 940 | L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS); | 923 | wa_write_or(wal, |
| 924 | SLICE_UNIT_LEVEL_CLKGATE, | ||
| 925 | L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS); | ||
| 926 | } | ||
| 941 | } | 927 | } |
| 942 | 928 | ||
| 943 | static void | 929 | static void |
| @@ -28,7 +28,6 @@ · 27 unmodified lines | |||
| 28 | #include "i915_drv.h" | 28 | #include "i915_drv.h" |
| 29 | #include "intel_context.h" | 29 | #include "intel_context.h" |
| 30 | #include "intel_engine_pm.h" | 30 | #include "intel_engine_pm.h" |
| 31 | #include "intel_engine_pool.h" | ||
| 32 | 31 | ||
| 33 | #include "mock_engine.h" | 32 | #include "mock_engine.h" |
| 34 | #include "selftests/mock_request.h" | 33 | #include "selftests/mock_request.h" |
| @@ -328,7 +327,6 @@ int mock_engine_init(struct intel_engine_cs *engine) · 293 unmodified lines | |||
| 328 | intel_engine_init_execlists(engine); | 327 | intel_engine_init_execlists(engine); |
| 329 | intel_engine_init__pm(engine); | 328 | intel_engine_init__pm(engine); |
| 330 | intel_engine_init_retire(engine); | 329 | intel_engine_init_retire(engine); |
| 331 | intel_engine_pool_init(&engine->pool); | ||
| 332 | 330 | ||
| 333 | ce = create_kernel_context(engine); | 331 | ce = create_kernel_context(engine); |
| 334 | if (IS_ERR(ce)) | 332 | if (IS_ERR(ce)) |
| @@ -24,6 +24,7 @@ static int request_sync(struct i915_request *rq) · 23 unmodified lines | |||
| 24 | 24 | ||
| 25 | /* Opencode i915_request_add() so we can keep the timeline locked. */ | 25 | /* Opencode i915_request_add() so we can keep the timeline locked. */ |
| 26 | __i915_request_commit(rq); | 26 | __i915_request_commit(rq); |
| 27 | rq->sched.attr.priority = I915_PRIORITY_BARRIER; | ||
| 27 | __i915_request_queue(rq, NULL); | 28 | __i915_request_queue(rq, NULL); |
| 28 | 29 | ||
| 29 | timeout = i915_request_wait(rq, 0, HZ / 10); | 30 | timeout = i915_request_wait(rq, 0, HZ / 10); |
| @@ -154,10 +155,7 @@ static int live_context_size(void *arg) · 124 unmodified lines | |||
| 154 | */ | 155 | */ |
| 155 | 156 | ||
| 156 | for_each_engine(engine, gt, id) { | 157 | for_each_engine(engine, gt, id) { |
| 157 | struct { | 158 | struct file *saved; |
| 158 | struct file *state; | ||
| 159 | void *pinned; | ||
| 160 | } saved; | ||
| 161 | 159 | ||
| 162 | if (!engine->context_size) | 160 | if (!engine->context_size) |
| 163 | continue; | 161 | continue; |
| @@ -171,8 +169,7 @@ static int live_context_size(void *arg) · 7 unmodified lines | |||
| 171 | * active state is sufficient, we are only checking that we | 169 | * active state is sufficient, we are only checking that we |
| 172 | * don't use more than we planned. | 170 | * don't use more than we planned. |
| 173 | */ | 171 | */ |
| 174 | saved.state = fetch_and_zero(&engine->default_state); | 172 | saved = fetch_and_zero(&engine->default_state); |
| 175 | saved.pinned = fetch_and_zero(&engine->pinned_default_state); | ||
| 176 | 173 | ||
| 177 | /* Overlaps with the execlists redzone */ | 174 | /* Overlaps with the execlists redzone */ |
| 178 | engine->context_size += I915_GTT_PAGE_SIZE; | 175 | engine->context_size += I915_GTT_PAGE_SIZE; |
| @@ -181,8 +178,7 @@ static int live_context_size(void *arg) · 2 unmodified lines | |||
| 181 | 178 | ||
| 182 | engine->context_size -= I915_GTT_PAGE_SIZE; | 179 | engine->context_size -= I915_GTT_PAGE_SIZE; |
| 183 | 180 | ||
| 184 | engine->pinned_default_state = saved.pinned; | 181 | engine->default_state = saved; |
| 185 | engine->default_state = saved.state; | ||
| 186 | 182 | ||
| 187 | intel_engine_pm_put(engine); | 183 | intel_engine_pm_put(engine); |
| 188 | 184 | ||
| @@ -4342,35 +4342,6 @@ int intel_execlists_live_selftests(struct drm_i915_private *i915) · 4341 unmodified lines | |||
| 4342 | return intel_gt_live_subtests(tests, &i915->gt); | 4342 | return intel_gt_live_subtests(tests, &i915->gt); |
| 4343 | } | 4343 | } |
| 4344 | 4344 | ||
| 4345 | static void hexdump(const void *buf, size_t len) | ||
| 4346 | { | ||
| 4347 | const size_t rowsize = 8 * sizeof(u32); | ||
| 4348 | const void *prev = NULL; | ||
| 4349 | bool skip = false; | ||
| 4350 | size_t pos; | ||
| 4351 | |||
| 4352 | for (pos = 0; pos < len; pos += rowsize) { | ||
| 4353 | char line[128]; | ||
| 4354 | |||
| 4355 | if (prev && !memcmp(prev, buf + pos, rowsize)) { | ||
| 4356 | if (!skip) { | ||
| 4357 | pr_info("*\n"); | ||
| 4358 | skip = true; | ||
| 4359 | } | ||
| 4360 | continue; | ||
| 4361 | } | ||
| 4362 | |||
| 4363 | WARN_ON_ONCE(hex_dump_to_buffer(buf + pos, len - pos, | ||
| 4364 | rowsize, sizeof(u32), | ||
| 4365 | line, sizeof(line), | ||
| 4366 | false) >= sizeof(line)); | ||
| 4367 | pr_info("[%04zx] %s\n", pos, line); | ||
| 4368 | |||
| 4369 | prev = buf + pos; | ||
| 4370 | skip = false; | ||
| 4371 | } | ||
| 4372 | } | ||
| 4373 | |||
| 4374 | static int emit_semaphore_signal(struct intel_context *ce, void *slot) | 4345 | static int emit_semaphore_signal(struct intel_context *ce, void *slot) |
| 4375 | { | 4346 | { |
| 4376 | const u32 offset = | 4347 | const u32 offset = |
| @@ -4518,10 +4489,10 @@ static int live_lrc_layout(void *arg) · 141 unmodified lines | |||
| 4518 | 4489 | ||
| 4519 | if (err) { | 4490 | if (err) { |
| 4520 | pr_info("%s: HW register image:\n", engine->name); | 4491 | pr_info("%s: HW register image:\n", engine->name); |
| 4521 | hexdump(hw, PAGE_SIZE); | 4492 | igt_hexdump(hw, PAGE_SIZE); |
| 4522 | 4493 | ||
| 4523 | pr_info("%s: SW register image:\n", engine->name); | 4494 | pr_info("%s: SW register image:\n", engine->name); |
| 4524 | hexdump(lrc, PAGE_SIZE); | 4495 | igt_hexdump(lrc, PAGE_SIZE); |
| 4525 | } | 4496 | } |
| 4526 | 4497 | ||
| 4527 | shmem_unpin_map(engine->default_state, hw); | 4498 | shmem_unpin_map(engine->default_state, hw); |
| @@ -5206,6 +5177,7 @@ store_context(struct intel_context *ce, struct i915_vma *scratch) · 678 unmodified lines | |||
| 5206 | { | 5177 | { |
| 5207 | struct i915_vma *batch; | 5178 | struct i915_vma *batch; |
| 5208 | u32 dw, x, *cs, *hw; | 5179 | u32 dw, x, *cs, *hw; |
| 5180 | u32 *defaults; | ||
| 5209 | 5181 | ||
| 5210 | batch = create_user_vma(ce->vm, SZ_64K); | 5182 | batch = create_user_vma(ce->vm, SZ_64K); |
| 5211 | if (IS_ERR(batch)) | 5183 | if (IS_ERR(batch)) |
| @@ -5217,9 +5189,16 @@ store_context(struct intel_context *ce, struct i915_vma *scratch) · 5 unmodified lines | |||
| 5217 | return ERR_CAST(cs); | 5189 | return ERR_CAST(cs); |
| 5218 | } | 5190 | } |
| 5219 | 5191 | ||
| 5192 | defaults = shmem_pin_map(ce->engine->default_state); | ||
| 5193 | if (!defaults) { | ||
| 5194 | i915_gem_object_unpin_map(batch->obj); | ||
| 5195 | i915_vma_put(batch); | ||
| 5196 | return ERR_PTR(-ENOMEM); | ||
| 5197 | } | ||
| 5198 | |||
| 5220 | x = 0; | 5199 | x = 0; |
| 5221 | dw = 0; | 5200 | dw = 0; |
| 5222 | hw = ce->engine->pinned_default_state; | 5201 | hw = defaults; |
| 5223 | hw += LRC_STATE_OFFSET / sizeof(*hw); | 5202 | hw += LRC_STATE_OFFSET / sizeof(*hw); |
| 5224 | do { | 5203 | do { |
| 5225 | u32 len = hw[dw] & 0x7f; | 5204 | u32 len = hw[dw] & 0x7f; |
| @@ -5250,6 +5229,8 @@ store_context(struct intel_context *ce, struct i915_vma *scratch) · 24 unmodified lines | |||
| 5250 | 5229 | ||
| 5251 | *cs++ = MI_BATCH_BUFFER_END; | 5230 | *cs++ = MI_BATCH_BUFFER_END; |
| 5252 | 5231 | ||
| 5232 | shmem_unpin_map(ce->engine->default_state, defaults); | ||
| 5233 | |||
| 5253 | i915_gem_object_flush_map(batch->obj); | 5234 | i915_gem_object_flush_map(batch->obj); |
| 5254 | i915_gem_object_unpin_map(batch->obj); | 5235 | i915_gem_object_unpin_map(batch->obj); |
| 5255 | 5236 | ||
| @@ -5360,6 +5341,7 @@ static struct i915_vma *load_context(struct intel_context *ce, u32 poison) · 104 unmodified lines | |||
| 5360 | { | 5341 | { |
| 5361 | struct i915_vma *batch; | 5342 | struct i915_vma *batch; |
| 5362 | u32 dw, *cs, *hw; | 5343 | u32 dw, *cs, *hw; |
| 5344 | u32 *defaults; | ||
| 5363 | 5345 | ||
| 5364 | batch = create_user_vma(ce->vm, SZ_64K); | 5346 | batch = create_user_vma(ce->vm, SZ_64K); |
| 5365 | if (IS_ERR(batch)) | 5347 | if (IS_ERR(batch)) |
| @@ -5371,8 +5353,15 @@ static struct i915_vma *load_context(struct intel_context *ce, u32 poison) · 5 unmodified lines | |||
| 5371 | return ERR_CAST(cs); | 5353 | return ERR_CAST(cs); |
| 5372 | } | 5354 | } |
| 5373 | 5355 | ||
| 5356 | defaults = shmem_pin_map(ce->engine->default_state); | ||
| 5357 | if (!defaults) { | ||
| 5358 | i915_gem_object_unpin_map(batch->obj); | ||
| 5359 | i915_vma_put(batch); | ||
| 5360 | return ERR_PTR(-ENOMEM); | ||
| 5361 | } | ||
| 5362 | |||
| 5374 | dw = 0; | 5363 | dw = 0; |
| 5375 | hw = ce->engine->pinned_default_state; | 5364 | hw = defaults; |
| 5376 | hw += LRC_STATE_OFFSET / sizeof(*hw); | 5365 | hw += LRC_STATE_OFFSET / sizeof(*hw); |
| 5377 | do { | 5366 | do { |
| 5378 | u32 len = hw[dw] & 0x7f; | 5367 | u32 len = hw[dw] & 0x7f; |
| @@ -5400,6 +5389,8 @@ static struct i915_vma *load_context(struct intel_context *ce, u32 poison) · 21 unmodified lines | |||
| 5400 | 5389 | ||
| 5401 | *cs++ = MI_BATCH_BUFFER_END; | 5390 | *cs++ = MI_BATCH_BUFFER_END; |
| 5402 | 5391 | ||
| 5392 | shmem_unpin_map(ce->engine->default_state, defaults); | ||
| 5393 | |||
| 5403 | i915_gem_object_flush_map(batch->obj); | 5394 | i915_gem_object_flush_map(batch->obj); |
| 5404 | i915_gem_object_unpin_map(batch->obj); | 5395 | i915_gem_object_unpin_map(batch->obj); |
| 5405 | 5396 | ||
| @@ -5467,6 +5458,7 @@ static int compare_isolation(struct intel_engine_cs *engine, · 61 unmodified lines | |||
| 5467 | { | 5458 | { |
| 5468 | u32 x, dw, *hw, *lrc; | 5459 | u32 x, dw, *hw, *lrc; |
| 5469 | u32 *A[2], *B[2]; | 5460 | u32 *A[2], *B[2]; |
| 5461 | u32 *defaults; | ||
| 5470 | int err = 0; | 5462 | int err = 0; |
| 5471 | 5463 | ||
| 5472 | A[0] = i915_gem_object_pin_map(ref[0]->obj, I915_MAP_WC); | 5464 | A[0] = i915_gem_object_pin_map(ref[0]->obj, I915_MAP_WC); |
| @@ -5499,9 +5491,15 @@ static int compare_isolation(struct intel_engine_cs *engine, · 26 unmodified lines | |||
| 5499 | } | 5491 | } |
| 5500 | lrc += LRC_STATE_OFFSET / sizeof(*hw); | 5492 | lrc += LRC_STATE_OFFSET / sizeof(*hw); |
| 5501 | 5493 | ||
| 5494 | defaults = shmem_pin_map(ce->engine->default_state); | ||
| 5495 | if (!defaults) { | ||
| 5496 | err = -ENOMEM; | ||
| 5497 | goto err_lrc; | ||
| 5498 | } | ||
| 5499 | |||
| 5502 | x = 0; | 5500 | x = 0; |
| 5503 | dw = 0; | 5501 | dw = 0; |
| 5504 | hw = engine->pinned_default_state; | 5502 | hw = defaults; |
| 5505 | hw += LRC_STATE_OFFSET / sizeof(*hw); | 5503 | hw += LRC_STATE_OFFSET / sizeof(*hw); |
| 5506 | do { | 5504 | do { |
| 5507 | u32 len = hw[dw] & 0x7f; | 5505 | u32 len = hw[dw] & 0x7f; |
| @@ -5541,6 +5539,8 @@ static int compare_isolation(struct intel_engine_cs *engine, · 33 unmodified lines | |||
| 5541 | } while (dw < PAGE_SIZE / sizeof(u32) && | 5539 | } while (dw < PAGE_SIZE / sizeof(u32) && |
| 5542 | (hw[dw] & ~BIT(0)) != MI_BATCH_BUFFER_END); | 5540 | (hw[dw] & ~BIT(0)) != MI_BATCH_BUFFER_END); |
| 5543 | 5541 | ||
| 5542 | shmem_unpin_map(ce->engine->default_state, defaults); | ||
| 5543 | err_lrc: | ||
| 5544 | i915_gem_object_unpin_map(ce->state->obj); | 5544 | i915_gem_object_unpin_map(ce->state->obj); |
| 5545 | err_B1: | 5545 | err_B1: |
| 5546 | i915_gem_object_unpin_map(result[1]->obj); | 5546 | i915_gem_object_unpin_map(result[1]->obj); |
| @@ -5690,18 +5690,16 @@ static int live_lrc_isolation(void *arg) · 143 unmodified lines | |||
| 5690 | continue; | 5690 | continue; |
| 5691 | 5691 | ||
| 5692 | intel_engine_pm_get(engine); | 5692 | intel_engine_pm_get(engine); |
| 5693 | if (engine->pinned_default_state) { | 5693 | for (i = 0; i < ARRAY_SIZE(poison); i++) { |
| 5694 | for (i = 0; i < ARRAY_SIZE(poison); i++) { | 5694 | int result; |
| 5695 | int result; | ||
| 5696 | 5695 | ||
| 5697 | result = __lrc_isolation(engine, poison[i]); | 5696 | result = __lrc_isolation(engine, poison[i]); |
| 5698 | if (result && !err) | 5697 | if (result && !err) |
| 5699 | err = result; | 5698 | err = result; |
| 5700 | 5699 | ||
| 5701 | result = __lrc_isolation(engine, ~poison[i]); | 5700 | result = __lrc_isolation(engine, ~poison[i]); |
| 5702 | if (result && !err) | 5701 | if (result && !err) |
| 5703 | err = result; | 5702 | err = result; |
| 5704 | } | ||
| 5705 | } | 5703 | } |
| 5706 | intel_engine_pm_put(engine); | 5704 | intel_engine_pm_put(engine); |
| 5707 | if (igt_flush_test(gt->i915)) { | 5705 | if (igt_flush_test(gt->i915)) { |
| @@ -54,6 +54,8 @@ static struct i915_vma *create_wally(struct intel_engine_cs *engine) · 53 unmodified lines | |||
| 54 | *cs++ = STACK_MAGIC; | 54 | *cs++ = STACK_MAGIC; |
| 55 | 55 | ||
| 56 | *cs++ = MI_BATCH_BUFFER_END; | 56 | *cs++ = MI_BATCH_BUFFER_END; |
| 57 | |||
| 58 | i915_gem_object_flush_map(obj); | ||
| 57 | i915_gem_object_unpin_map(obj); | 59 | i915_gem_object_unpin_map(obj); |
| 58 | 60 | ||
| 59 | vma->private = intel_context_create(engine); /* dummy residuals */ | 61 | vma->private = intel_context_create(engine); /* dummy residuals */ |
| @@ -56,6 +56,18 @@ static int cmp_u64(const void *A, const void *B) · 55 unmodified lines | |||
| 56 | return 0; | 56 | return 0; |
| 57 | } | 57 | } |
| 58 | 58 | ||
| 59 | static int cmp_u32(const void *A, const void *B) | ||
| 60 | { | ||
| 61 | const u32 *a = A, *b = B; | ||
| 62 | |||
| 63 | if (a < b) | ||
| 64 | return -1; | ||
| 65 | else if (a > b) | ||
| 66 | return 1; | ||
| 67 | else | ||
| 68 | return 0; | ||
| 69 | } | ||
| 70 | |||
| 59 | static struct i915_vma * | 71 | static struct i915_vma * |
| 60 | create_spin_counter(struct intel_engine_cs *engine, | 72 | create_spin_counter(struct intel_engine_cs *engine, |
| 61 | struct i915_address_space *vm, | 73 | struct i915_address_space *vm, |
| @@ -236,8 +248,8 @@ int live_rps_clock_interval(void *arg) · 174 unmodified lines | |||
| 236 | for_each_engine(engine, gt, id) { | 248 | for_each_engine(engine, gt, id) { |
| 237 | unsigned long saved_heartbeat; | 249 | unsigned long saved_heartbeat; |
| 238 | struct i915_request *rq; | 250 | struct i915_request *rq; |
| 239 | ktime_t dt; | ||
| 240 | u32 cycles; | 251 | u32 cycles; |
| 252 | u64 dt; | ||
| 241 | 253 | ||
| 242 | if (!intel_engine_can_store_dword(engine)) | 254 | if (!intel_engine_can_store_dword(engine)) |
| 243 | continue; | 255 | continue; |
| @@ -286,15 +298,29 @@ int live_rps_clock_interval(void *arg) · 42 unmodified lines | |||
| 286 | engine->name); | 298 | engine->name); |
| 287 | err = -ENODEV; | 299 | err = -ENODEV; |
| 288 | } else { | 300 | } else { |
| 289 | preempt_disable(); | 301 | ktime_t dt_[5]; |
| 290 | dt = ktime_get(); | 302 | u32 cycles_[5]; |
| 291 | cycles = -intel_uncore_read_fw(gt->uncore, | 303 | int i; |
| 292 | GEN6_RP_CUR_UP_EI); | 304 | |
| 293 | udelay(1000); | 305 | for (i = 0; i < 5; i++) { |
| 294 | dt = ktime_sub(ktime_get(), dt); | 306 | preempt_disable(); |
| 295 | cycles += intel_uncore_read_fw(gt->uncore, | 307 | |
| 296 | GEN6_RP_CUR_UP_EI); | 308 | dt_[i] = ktime_get(); |
| 297 | preempt_enable(); | 309 | cycles_[i] = -intel_uncore_read_fw(gt->uncore, GEN6_RP_CUR_UP_EI); |
| 310 | |||
| 311 | udelay(1000); | ||
| 312 | |||
| 313 | dt_[i] = ktime_sub(ktime_get(), dt_[i]); | ||
| 314 | cycles_[i] += intel_uncore_read_fw(gt->uncore, GEN6_RP_CUR_UP_EI); | ||
| 315 | |||
| 316 | preempt_enable(); | ||
| 317 | } | ||
| 318 | |||
| 319 | /* Use the median of both cycle/dt; close enough */ | ||
| 320 | sort(cycles_, 5, sizeof(*cycles_), cmp_u32, NULL); | ||
| 321 | cycles = (cycles_[1] + 2 * cycles_[2] + cycles_[3]) / 4; | ||
| 322 | sort(dt_, 5, sizeof(*dt_), cmp_u64, NULL); | ||
| 323 | dt = div_u64(dt_[1] + 2 * dt_[2] + dt_[3], 4); | ||
| 298 | } | 324 | } |
| 299 | 325 | ||
| 300 | intel_uncore_write_fw(gt->uncore, GEN6_RP_CONTROL, 0); | 326 | intel_uncore_write_fw(gt->uncore, GEN6_RP_CONTROL, 0); |
| @@ -306,14 +332,14 @@ int live_rps_clock_interval(void *arg) · 5 unmodified lines | |||
| 306 | if (err == 0) { | 332 | if (err == 0) { |
| 307 | u64 time = intel_gt_pm_interval_to_ns(gt, cycles); | 333 | u64 time = intel_gt_pm_interval_to_ns(gt, cycles); |
| 308 | u32 expected = | 334 | u32 expected = |
| 309 | intel_gt_ns_to_pm_interval(gt, ktime_to_ns(dt)); | 335 | intel_gt_ns_to_pm_interval(gt, dt); |
| 310 | 336 | ||
| 311 | pr_info("%s: rps counted %d C0 cycles [%lldns] in %lldns [%d cycles], using GT clock frequency of %uKHz\n", | 337 | pr_info("%s: rps counted %d C0 cycles [%lldns] in %lldns [%d cycles], using GT clock frequency of %uKHz\n", |
| 312 | engine->name, cycles, time, ktime_to_ns(dt), expected, | 338 | engine->name, cycles, time, dt, expected, |
| 313 | gt->clock_frequency / 1000); | 339 | gt->clock_frequency / 1000); |
| 314 | 340 | ||
| 315 | if (10 * time < 8 * ktime_to_ns(dt) || | 341 | if (10 * time < 8 * dt || |
| 316 | 8 * time > 10 * ktime_to_ns(dt)) { | 342 | 8 * time > 10 * dt) { |
| 317 | pr_err("%s: rps clock time does not match walltime!\n", | 343 | pr_err("%s: rps clock time does not match walltime!\n", |
| 318 | engine->name); | 344 | engine->name); |
| 319 | err = -EINVAL; | 345 | err = -EINVAL; |
| @@ -701,6 +727,7 @@ int live_rps_frequency_cs(void *arg) · 381 unmodified lines | |||
| 701 | 727 | ||
| 702 | err_vma: | 728 | err_vma: |
| 703 | *cancel = MI_BATCH_BUFFER_END; | 729 | *cancel = MI_BATCH_BUFFER_END; |
| 730 | i915_gem_object_flush_map(vma->obj); | ||
| 704 | i915_gem_object_unpin_map(vma->obj); | 731 | i915_gem_object_unpin_map(vma->obj); |
| 705 | i915_vma_unpin(vma); | 732 | i915_vma_unpin(vma); |
| 706 | i915_vma_put(vma); | 733 | i915_vma_put(vma); |
| @@ -842,6 +869,7 @@ int live_rps_frequency_srm(void *arg) · 135 unmodified lines | |||
| 842 | 869 | ||
| 843 | err_vma: | 870 | err_vma: |
| 844 | *cancel = MI_BATCH_BUFFER_END; | 871 | *cancel = MI_BATCH_BUFFER_END; |
| 872 | i915_gem_object_flush_map(vma->obj); | ||
| 845 | i915_gem_object_unpin_map(vma->obj); | 873 | i915_gem_object_unpin_map(vma->obj); |
| 846 | i915_vma_unpin(vma); | 874 | i915_vma_unpin(vma); |
| 847 | i915_vma_put(vma); | 875 | i915_vma_put(vma); |
| @@ -191,6 +191,17 @@ max_spin_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) · 190 unmodified lines | |||
| 191 | static struct kobj_attribute max_spin_attr = | 191 | static struct kobj_attribute max_spin_attr = |
| 192 | __ATTR(max_busywait_duration_ns, 0644, max_spin_show, max_spin_store); | 192 | __ATTR(max_busywait_duration_ns, 0644, max_spin_show, max_spin_store); |
| 193 | 193 | ||
| 194 | static ssize_t | ||
| 195 | max_spin_default(struct kobject *kobj, struct kobj_attribute *attr, char *buf) | ||
| 196 | { | ||
| 197 | struct intel_engine_cs *engine = kobj_to_engine(kobj); | ||
| 198 | |||
| 199 | return sprintf(buf, "%lu\n", engine->defaults.max_busywait_duration_ns); | ||
| 200 | } | ||
| 201 | |||
| 202 | static struct kobj_attribute max_spin_def = | ||
| 203 | __ATTR(max_busywait_duration_ns, 0444, max_spin_default, NULL); | ||
| 204 | |||
| 194 | static ssize_t | 205 | static ssize_t |
| 195 | timeslice_store(struct kobject *kobj, struct kobj_attribute *attr, | 206 | timeslice_store(struct kobject *kobj, struct kobj_attribute *attr, |
| 196 | const char *buf, size_t count) | 207 | const char *buf, size_t count) |
| @@ -233,6 +244,17 @@ timeslice_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) · 36 unmodified lines | |||
| 233 | static struct kobj_attribute timeslice_duration_attr = | 244 | static struct kobj_attribute timeslice_duration_attr = |
| 234 | __ATTR(timeslice_duration_ms, 0644, timeslice_show, timeslice_store); | 245 | __ATTR(timeslice_duration_ms, 0644, timeslice_show, timeslice_store); |
| 235 | 246 | ||
| 247 | static ssize_t | ||
| 248 | timeslice_default(struct kobject *kobj, struct kobj_attribute *attr, char *buf) | ||
| 249 | { | ||
| 250 | struct intel_engine_cs *engine = kobj_to_engine(kobj); | ||
| 251 | |||
| 252 | return sprintf(buf, "%lu\n", engine->defaults.timeslice_duration_ms); | ||
| 253 | } | ||
| 254 | |||
| 255 | static struct kobj_attribute timeslice_duration_def = | ||
| 256 | __ATTR(timeslice_duration_ms, 0444, timeslice_default, NULL); | ||
| 257 | |||
| 236 | static ssize_t | 258 | static ssize_t |
| 237 | stop_store(struct kobject *kobj, struct kobj_attribute *attr, | 259 | stop_store(struct kobject *kobj, struct kobj_attribute *attr, |
| 238 | const char *buf, size_t count) | 260 | const char *buf, size_t count) |
| @@ -272,6 +294,17 @@ stop_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) · 33 unmodified lines | |||
| 272 | static struct kobj_attribute stop_timeout_attr = | 294 | static struct kobj_attribute stop_timeout_attr = |
| 273 | __ATTR(stop_timeout_ms, 0644, stop_show, stop_store); | 295 | __ATTR(stop_timeout_ms, 0644, stop_show, stop_store); |
| 274 | 296 | ||
| 297 | static ssize_t | ||
| 298 | stop_default(struct kobject *kobj, struct kobj_attribute *attr, char *buf) | ||
| 299 | { | ||
| 300 | struct intel_engine_cs *engine = kobj_to_engine(kobj); | ||
| 301 | |||
| 302 | return sprintf(buf, "%lu\n", engine->defaults.stop_timeout_ms); | ||
| 303 | } | ||
| 304 | |||
| 305 | static struct kobj_attribute stop_timeout_def = | ||
| 306 | __ATTR(stop_timeout_ms, 0444, stop_default, NULL); | ||
| 307 | |||
| 275 | static ssize_t | 308 | static ssize_t |
| 276 | preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr, | 309 | preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr, |
| 277 | const char *buf, size_t count) | 310 | const char *buf, size_t count) |
| @@ -316,6 +349,18 @@ preempt_timeout_show(struct kobject *kobj, struct kobj_attribute *attr, · 38 unmodified lines | |||
| 316 | static struct kobj_attribute preempt_timeout_attr = | 349 | static struct kobj_attribute preempt_timeout_attr = |
| 317 | __ATTR(preempt_timeout_ms, 0644, preempt_timeout_show, preempt_timeout_store); | 350 | __ATTR(preempt_timeout_ms, 0644, preempt_timeout_show, preempt_timeout_store); |
| 318 | 351 | ||
| 352 | static ssize_t | ||
| 353 | preempt_timeout_default(struct kobject *kobj, struct kobj_attribute *attr, | ||
| 354 | char *buf) | ||
| 355 | { | ||
| 356 | struct intel_engine_cs *engine = kobj_to_engine(kobj); | ||
| 357 | |||
| 358 | return sprintf(buf, "%lu\n", engine->defaults.preempt_timeout_ms); | ||
| 359 | } | ||
| 360 | |||
| 361 | static struct kobj_attribute preempt_timeout_def = | ||
| 362 | __ATTR(preempt_timeout_ms, 0444, preempt_timeout_default, NULL); | ||
| 363 | |||
| 319 | static ssize_t | 364 | static ssize_t |
| 320 | heartbeat_store(struct kobject *kobj, struct kobj_attribute *attr, | 365 | heartbeat_store(struct kobject *kobj, struct kobj_attribute *attr, |
| 321 | const char *buf, size_t count) | 366 | const char *buf, size_t count) |
| @@ -359,6 +404,17 @@ heartbeat_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) · 37 unmodified lines | |||
| 359 | static struct kobj_attribute heartbeat_interval_attr = | 404 | static struct kobj_attribute heartbeat_interval_attr = |
| 360 | __ATTR(heartbeat_interval_ms, 0644, heartbeat_show, heartbeat_store); | 405 | __ATTR(heartbeat_interval_ms, 0644, heartbeat_show, heartbeat_store); |
| 361 | 406 | ||
| 407 | static ssize_t | ||
| 408 | heartbeat_default(struct kobject *kobj, struct kobj_attribute *attr, char *buf) | ||
| 409 | { | ||
| 410 | struct intel_engine_cs *engine = kobj_to_engine(kobj); | ||
| 411 | |||
| 412 | return sprintf(buf, "%lu\n", engine->defaults.heartbeat_interval_ms); | ||
| 413 | } | ||
| 414 | |||
| 415 | static struct kobj_attribute heartbeat_interval_def = | ||
| 416 | __ATTR(heartbeat_interval_ms, 0444, heartbeat_default, NULL); | ||
| 417 | |||
| 362 | static void kobj_engine_release(struct kobject *kobj) | 418 | static void kobj_engine_release(struct kobject *kobj) |
| 363 | { | 419 | { |
| 364 | kfree(kobj); | 420 | kfree(kobj); |
| @@ -390,6 +446,42 @@ kobj_engine(struct kobject *dir, struct intel_engine_cs *engine) · 25 unmodified lines | |||
| 390 | return &ke->base; | 446 | return &ke->base; |
| 391 | } | 447 | } |
| 392 | 448 | ||
| 449 | static void add_defaults(struct kobj_engine *parent) | ||
| 450 | { | ||
| 451 | static const struct attribute *files[] = { | ||
| 452 | &max_spin_def.attr, | ||
| 453 | &stop_timeout_def.attr, | ||
| 454 | #if CONFIG_DRM_I915_HEARTBEAT_INTERVAL | ||
| 455 | &heartbeat_interval_def.attr, | ||
| 456 | #endif | ||
| 457 | NULL | ||
| 458 | }; | ||
| 459 | struct kobj_engine *ke; | ||
| 460 | |||
| 461 | ke = kzalloc(sizeof(*ke), GFP_KERNEL); | ||
| 462 | if (!ke) | ||
| 463 | return; | ||
| 464 | |||
| 465 | kobject_init(&ke->base, &kobj_engine_type); | ||
| 466 | ke->engine = parent->engine; | ||
| 467 | |||
| 468 | if (kobject_add(&ke->base, &parent->base, "%s", ".defaults")) { | ||
| 469 | kobject_put(&ke->base); | ||
| 470 | return; | ||
| 471 | } | ||
| 472 | |||
| 473 | if (sysfs_create_files(&ke->base, files)) | ||
| 474 | return; | ||
| 475 | |||
| 476 | if (intel_engine_has_timeslices(ke->engine) && | ||
| 477 | sysfs_create_file(&ke->base, ×lice_duration_def.attr)) | ||
| 478 | return; | ||
| 479 | |||
| 480 | if (intel_engine_has_preempt_reset(ke->engine) && | ||
| 481 | sysfs_create_file(&ke->base, &preempt_timeout_def.attr)) | ||
| 482 | return; | ||
| 483 | } | ||
| 484 | |||
| 393 | void intel_engines_add_sysfs(struct drm_i915_private *i915) | 485 | void intel_engines_add_sysfs(struct drm_i915_private *i915) |
| 394 | { | 486 | { |
| 395 | static const struct attribute *files[] = { | 487 | static const struct attribute *files[] = { |
| @@ -433,6 +525,8 @@ void intel_engines_add_sysfs(struct drm_i915_private *i915) · 37 unmodified lines | |||
| 433 | sysfs_create_file(kobj, &preempt_timeout_attr.attr)) | 525 | sysfs_create_file(kobj, &preempt_timeout_attr.attr)) |
| 434 | goto err_engine; | 526 | goto err_engine; |
| 435 | 527 | ||
| 528 | add_defaults(container_of(kobj, struct kobj_engine, base)); | ||
| 529 | |||
| 436 | if (0) { | 530 | if (0) { |
| 437 | err_object: | 531 | err_object: |
| 438 | kobject_put(kobj); | 532 | kobject_put(kobj); |
| @@ -258,7 +258,7 @@ static void guc_submit(struct intel_engine_cs *engine, · 257 unmodified lines | |||
| 258 | 258 | ||
| 259 | static inline int rq_prio(const struct i915_request *rq) | 259 | static inline int rq_prio(const struct i915_request *rq) |
| 260 | { | 260 | { |
| 261 | return rq->sched.attr.priority | __NO_PREEMPTION; | 261 | return rq->sched.attr.priority; |
| 262 | } | 262 | } |
| 263 | 263 | ||
| 264 | static struct i915_request *schedule_in(struct i915_request *rq, int idx) | 264 | static struct i915_request *schedule_in(struct i915_request *rq, int idx) |
| @@ -882,6 +882,47 @@ static int mocs_cmd_reg_handler(struct parser_exec_state *s, · 881 unmodified lines | |||
| 882 | return 0; | 882 | return 0; |
| 883 | } | 883 | } |
| 884 | 884 | ||
| 885 | static int is_cmd_update_pdps(unsigned int offset, | ||
| 886 | struct parser_exec_state *s) | ||
| 887 | { | ||
| 888 | u32 base = s->workload->engine->mmio_base; | ||
| 889 | return i915_mmio_reg_equal(_MMIO(offset), GEN8_RING_PDP_UDW(base, 0)); | ||
| 890 | } | ||
| 891 | |||
| 892 | static int cmd_pdp_mmio_update_handler(struct parser_exec_state *s, | ||
| 893 | unsigned int offset, unsigned int index) | ||
| 894 | { | ||
| 895 | struct intel_vgpu *vgpu = s->vgpu; | ||
| 896 | struct intel_vgpu_mm *shadow_mm = s->workload->shadow_mm; | ||
| 897 | struct intel_vgpu_mm *mm; | ||
| 898 | u64 pdps[GEN8_3LVL_PDPES]; | ||
| 899 | |||
| 900 | if (shadow_mm->ppgtt_mm.root_entry_type == | ||
| 901 | GTT_TYPE_PPGTT_ROOT_L4_ENTRY) { | ||
| 902 | pdps[0] = (u64)cmd_val(s, 2) << 32; | ||
| 903 | pdps[0] |= cmd_val(s, 4); | ||
| 904 | |||
| 905 | mm = intel_vgpu_find_ppgtt_mm(vgpu, pdps); | ||
| 906 | if (!mm) { | ||
| 907 | gvt_vgpu_err("failed to get the 4-level shadow vm\n"); | ||
| 908 | return -EINVAL; | ||
| 909 | } | ||
| 910 | intel_vgpu_mm_get(mm); | ||
| 911 | list_add_tail(&mm->ppgtt_mm.link, | ||
| 912 | &s->workload->lri_shadow_mm); | ||
| 913 | *cmd_ptr(s, 2) = upper_32_bits(mm->ppgtt_mm.shadow_pdps[0]); | ||
| 914 | *cmd_ptr(s, 4) = lower_32_bits(mm->ppgtt_mm.shadow_pdps[0]); | ||
| 915 | } else { | ||
| 916 | /* Currently all guests use PML4 table and now can't | ||
| 917 | * have a guest with 3-level table but uses LRI for | ||
| 918 | * PPGTT update. So this is simply un-testable. */ | ||
| 919 | GEM_BUG_ON(1); | ||
| 920 | gvt_vgpu_err("invalid shared shadow vm type\n"); | ||
| 921 | return -EINVAL; | ||
| 922 | } | ||
| 923 | return 0; | ||
| 924 | } | ||
| 925 | |||
| 885 | static int cmd_reg_handler(struct parser_exec_state *s, | 926 | static int cmd_reg_handler(struct parser_exec_state *s, |
| 886 | unsigned int offset, unsigned int index, char *cmd) | 927 | unsigned int offset, unsigned int index, char *cmd) |
| 887 | { | 928 | { |
| @@ -920,6 +961,10 @@ static int cmd_reg_handler(struct parser_exec_state *s, · 32 unmodified lines | |||
| 920 | patch_value(s, cmd_ptr(s, index), VGT_PVINFO_PAGE); | 961 | patch_value(s, cmd_ptr(s, index), VGT_PVINFO_PAGE); |
| 921 | } | 962 | } |
| 922 | 963 | ||
| 964 | if (is_cmd_update_pdps(offset, s) && | ||
| 965 | cmd_pdp_mmio_update_handler(s, offset, index)) | ||
| 966 | return -EINVAL; | ||
| 967 | |||
| 923 | /* TODO | 968 | /* TODO |
| 924 | * In order to let workload with inhibit context to generate | 969 | * In order to let workload with inhibit context to generate |
| 925 | * correct image data into memory, vregs values will be loaded to | 970 | * correct image data into memory, vregs values will be loaded to |
| @@ -424,8 +424,6 @@ static int complete_execlist_workload(struct intel_vgpu_workload *workload) · 423 unmodified lines | |||
| 424 | 424 | ||
| 425 | ret = emulate_execlist_ctx_schedule_out(execlist, &workload->ctx_desc); | 425 | ret = emulate_execlist_ctx_schedule_out(execlist, &workload->ctx_desc); |
| 426 | out: | 426 | out: |
| 427 | intel_vgpu_unpin_mm(workload->shadow_mm); | ||
| 428 | intel_vgpu_destroy_workload(workload); | ||
| 429 | return ret; | 427 | return ret; |
| 430 | } | 428 | } |
| 431 | 429 | ||
| @@ -1900,6 +1900,7 @@ struct intel_vgpu_mm *intel_vgpu_create_ppgtt_mm(struct intel_vgpu *vgpu, · 1899 unmodified lines | |||
| 1900 | 1900 | ||
| 1901 | INIT_LIST_HEAD(&mm->ppgtt_mm.list); | 1901 | INIT_LIST_HEAD(&mm->ppgtt_mm.list); |
| 1902 | INIT_LIST_HEAD(&mm->ppgtt_mm.lru_list); | 1902 | INIT_LIST_HEAD(&mm->ppgtt_mm.lru_list); |
| 1903 | INIT_LIST_HEAD(&mm->ppgtt_mm.link); | ||
| 1903 | 1904 | ||
| 1904 | if (root_entry_type == GTT_TYPE_PPGTT_ROOT_L4_ENTRY) | 1905 | if (root_entry_type == GTT_TYPE_PPGTT_ROOT_L4_ENTRY) |
| 1905 | mm->ppgtt_mm.guest_pdps[0] = pdps[0]; | 1906 | mm->ppgtt_mm.guest_pdps[0] = pdps[0]; |
| @@ -160,6 +160,7 @@ struct intel_vgpu_mm { · 159 unmodified lines | |||
| 160 | 160 | ||
| 161 | struct list_head list; | 161 | struct list_head list; |
| 162 | struct list_head lru_list; | 162 | struct list_head lru_list; |
| 163 | struct list_head link; /* possible LRI shadow mm list */ | ||
| 163 | } ppgtt_mm; | 164 | } ppgtt_mm; |
| 164 | struct { | 165 | struct { |
| 165 | void *virtual_ggtt; | 166 | void *virtual_ggtt; |
| @@ -2812,7 +2812,7 @@ static int init_bdw_mmio_info(struct intel_gvt *gvt) · 2811 unmodified lines | |||
| 2812 | MMIO_D(GAMTARBMODE, D_BDW_PLUS); | 2812 | MMIO_D(GAMTARBMODE, D_BDW_PLUS); |
| 2813 | 2813 | ||
| 2814 | #define RING_REG(base) _MMIO((base) + 0x270) | 2814 | #define RING_REG(base) _MMIO((base) + 0x270) |
| 2815 | MMIO_RING_F(RING_REG, 32, 0, 0, 0, D_BDW_PLUS, NULL, NULL); | 2815 | MMIO_RING_F(RING_REG, 32, F_CMD_ACCESS, 0, 0, D_BDW_PLUS, NULL, NULL); |
| 2816 | #undef RING_REG | 2816 | #undef RING_REG |
| 2817 | 2817 | ||
| 2818 | MMIO_RING_GM_RDR(RING_HWS_PGA, D_BDW_PLUS, NULL, hws_pga_write); | 2818 | MMIO_RING_GM_RDR(RING_HWS_PGA, D_BDW_PLUS, NULL, hws_pga_write); |
| @@ -58,10 +58,8 @@ static void set_context_pdp_root_pointer( · 57 unmodified lines | |||
| 58 | 58 | ||
| 59 | static void update_shadow_pdps(struct intel_vgpu_workload *workload) | 59 | static void update_shadow_pdps(struct intel_vgpu_workload *workload) |
| 60 | { | 60 | { |
| 61 | struct drm_i915_gem_object *ctx_obj = | ||
| 62 | workload->req->context->state->obj; | ||
| 63 | struct execlist_ring_context *shadow_ring_context; | 61 | struct execlist_ring_context *shadow_ring_context; |
| 64 | struct page *page; | 62 | struct intel_context *ctx = workload->req->context; |
| 65 | 63 | ||
| 66 | if (WARN_ON(!workload->shadow_mm)) | 64 | if (WARN_ON(!workload->shadow_mm)) |
| 67 | return; | 65 | return; |
| @@ -69,11 +67,9 @@ static void update_shadow_pdps(struct intel_vgpu_workload *workload) · 1 unmodified line | |||
| 69 | if (WARN_ON(!atomic_read(&workload->shadow_mm->pincount))) | 67 | if (WARN_ON(!atomic_read(&workload->shadow_mm->pincount))) |
| 70 | return; | 68 | return; |
| 71 | 69 | ||
| 72 | page = i915_gem_object_get_page(ctx_obj, LRC_STATE_PN); | 70 | shadow_ring_context = (struct execlist_ring_context *)ctx->lrc_reg_state; |
| 73 | shadow_ring_context = kmap(page); | ||
| 74 | set_context_pdp_root_pointer(shadow_ring_context, | 71 | set_context_pdp_root_pointer(shadow_ring_context, |
| 75 | (void *)workload->shadow_mm->ppgtt_mm.shadow_pdps); | 72 | (void *)workload->shadow_mm->ppgtt_mm.shadow_pdps); |
| 76 | kunmap(page); | ||
| 77 | } | 73 | } |
| 78 | 74 | ||
| 79 | /* | 75 | /* |
| @@ -646,10 +642,11 @@ static void release_shadow_batch_buffer(struct intel_vgpu_workload *workload) · 566 unmodified lines | |||
| 646 | } | 642 | } |
| 647 | } | 643 | } |
| 648 | 644 | ||
| 649 | static int prepare_workload(struct intel_vgpu_workload *workload) | 645 | static int |
| 646 | intel_vgpu_shadow_mm_pin(struct intel_vgpu_workload *workload) | ||
| 650 | { | 647 | { |
| 651 | struct intel_vgpu *vgpu = workload->vgpu; | 648 | struct intel_vgpu *vgpu = workload->vgpu; |
| 652 | struct intel_vgpu_submission *s = &vgpu->submission; | 649 | struct intel_vgpu_mm *m; |
| 653 | int ret = 0; | 650 | int ret = 0; |
| 654 | 651 | ||
| 655 | ret = intel_vgpu_pin_mm(workload->shadow_mm); | 652 | ret = intel_vgpu_pin_mm(workload->shadow_mm); |
| @@ -664,6 +661,52 @@ static int prepare_workload(struct intel_vgpu_workload *workload) · 8 unmodified lines | |||
| 664 | return -EINVAL; | 661 | return -EINVAL; |
| 665 | } | 662 | } |
| 666 | 663 | ||
| 664 | if (!list_empty(&workload->lri_shadow_mm)) { | ||
| 665 | list_for_each_entry(m, &workload->lri_shadow_mm, | ||
| 666 | ppgtt_mm.link) { | ||
| 667 | ret = intel_vgpu_pin_mm(m); | ||
| 668 | if (ret) { | ||
| 669 | list_for_each_entry_from_reverse(m, | ||
| 670 | &workload->lri_shadow_mm, | ||
| 671 | ppgtt_mm.link) | ||
| 672 | intel_vgpu_unpin_mm(m); | ||
| 673 | gvt_vgpu_err("LRI shadow ppgtt fail to pin\n"); | ||
| 674 | break; | ||
| 675 | } | ||
| 676 | } | ||
| 677 | } | ||
| 678 | |||
| 679 | if (ret) | ||
| 680 | intel_vgpu_unpin_mm(workload->shadow_mm); | ||
| 681 | |||
| 682 | return ret; | ||
| 683 | } | ||
| 684 | |||
| 685 | static void | ||
| 686 | intel_vgpu_shadow_mm_unpin(struct intel_vgpu_workload *workload) | ||
| 687 | { | ||
| 688 | struct intel_vgpu_mm *m; | ||
| 689 | |||
| 690 | if (!list_empty(&workload->lri_shadow_mm)) { | ||
| 691 | list_for_each_entry(m, &workload->lri_shadow_mm, | ||
| 692 | ppgtt_mm.link) | ||
| 693 | intel_vgpu_unpin_mm(m); | ||
| 694 | } | ||
| 695 | intel_vgpu_unpin_mm(workload->shadow_mm); | ||
| 696 | } | ||
| 697 | |||
| 698 | static int prepare_workload(struct intel_vgpu_workload *workload) | ||
| 699 | { | ||
| 700 | struct intel_vgpu *vgpu = workload->vgpu; | ||
| 701 | struct intel_vgpu_submission *s = &vgpu->submission; | ||
| 702 | int ret = 0; | ||
| 703 | |||
| 704 | ret = intel_vgpu_shadow_mm_pin(workload); | ||
| 705 | if (ret) { | ||
| 706 | gvt_vgpu_err("fail to pin shadow mm\n"); | ||
| 707 | return ret; | ||
| 708 | } | ||
| 709 | |||
| 667 | update_shadow_pdps(workload); | 710 | update_shadow_pdps(workload); |
| 668 | 711 | ||
| 669 | set_context_ppgtt_from_shadow(workload, s->shadow[workload->engine->id]); | 712 | set_context_ppgtt_from_shadow(workload, s->shadow[workload->engine->id]); |
| @@ -710,7 +753,7 @@ err_shadow_wa_ctx: · 40 unmodified lines | |||
| 710 | err_shadow_batch: | 753 | err_shadow_batch: |
| 711 | release_shadow_batch_buffer(workload); | 754 | release_shadow_batch_buffer(workload); |
| 712 | err_unpin_mm: | 755 | err_unpin_mm: |
| 713 | intel_vgpu_unpin_mm(workload->shadow_mm); | 756 | intel_vgpu_shadow_mm_unpin(workload); |
| 714 | return ret; | 757 | return ret; |
| 715 | } | 758 | } |
| 716 | 759 | ||
| @@ -820,6 +863,37 @@ out: · 103 unmodified lines | |||
| 820 | return workload; | 863 | return workload; |
| 821 | } | 864 | } |
| 822 | 865 | ||
| 866 | static void update_guest_pdps(struct intel_vgpu *vgpu, | ||
| 867 | u64 ring_context_gpa, u32 pdp[8]) | ||
| 868 | { | ||
| 869 | u64 gpa; | ||
| 870 | int i; | ||
| 871 | |||
| 872 | gpa = ring_context_gpa + RING_CTX_OFF(pdps[0].val); | ||
| 873 | |||
| 874 | for (i = 0; i < 8; i++) | ||
| 875 | intel_gvt_hypervisor_write_gpa(vgpu, | ||
| 876 | gpa + i * 8, &pdp[7 - i], 4); | ||
| 877 | } | ||
| 878 | |||
| 879 | static bool | ||
| 880 | check_shadow_context_ppgtt(struct execlist_ring_context *c, struct intel_vgpu_mm *m) | ||
| 881 | { | ||
| 882 | if (m->ppgtt_mm.root_entry_type == GTT_TYPE_PPGTT_ROOT_L4_ENTRY) { | ||
| 883 | u64 shadow_pdp = c->pdps[7].val | (u64) c->pdps[6].val << 32; | ||
| 884 | |||
| 885 | if (shadow_pdp != m->ppgtt_mm.shadow_pdps[0]) { | ||
| 886 | gvt_dbg_mm("4-level context ppgtt not match LRI command\n"); | ||
| 887 | return false; | ||
| 888 | } | ||
| 889 | return true; | ||
| 890 | } else { | ||
| 891 | /* see comment in LRI handler in cmd_parser.c */ | ||
| 892 | gvt_dbg_mm("invalid shadow mm type\n"); | ||
| 893 | return false; | ||
| 894 | } | ||
| 895 | } | ||
| 896 | |||
| 823 | static void update_guest_context(struct intel_vgpu_workload *workload) | 897 | static void update_guest_context(struct intel_vgpu_workload *workload) |
| 824 | { | 898 | { |
| 825 | struct i915_request *rq = workload->req; | 899 | struct i915_request *rq = workload->req; |
| @@ -905,6 +979,15 @@ write: · 79 unmodified lines | |||
| 905 | 979 | ||
| 906 | shadow_ring_context = (void *) ctx->lrc_reg_state; | 980 | shadow_ring_context = (void *) ctx->lrc_reg_state; |
| 907 | 981 | ||
| 982 | if (!list_empty(&workload->lri_shadow_mm)) { | ||
| 983 | struct intel_vgpu_mm *m = list_last_entry(&workload->lri_shadow_mm, | ||
| 984 | struct intel_vgpu_mm, | ||
| 985 | ppgtt_mm.link); | ||
| 986 | GEM_BUG_ON(!check_shadow_context_ppgtt(shadow_ring_context, m)); | ||
| 987 | update_guest_pdps(vgpu, workload->ring_context_gpa, | ||
| 988 | (void *)m->ppgtt_mm.guest_pdps); | ||
| 989 | } | ||
| 990 | |||
| 908 | #define COPY_REG(name) \ | 991 | #define COPY_REG(name) \ |
| 909 | intel_gvt_hypervisor_write_gpa(vgpu, workload->ring_context_gpa + \ | 992 | intel_gvt_hypervisor_write_gpa(vgpu, workload->ring_context_gpa + \ |
| 910 | RING_CTX_OFF(name.val), &shadow_ring_context->name.val, 4) | 993 | RING_CTX_OFF(name.val), &shadow_ring_context->name.val, 4) |
| @@ -1013,6 +1096,9 @@ static void complete_current_workload(struct intel_gvt *gvt, int ring_id) · 102 unmodified lines | |||
| 1013 | 1096 | ||
| 1014 | workload->complete(workload); | 1097 | workload->complete(workload); |
| 1015 | 1098 | ||
| 1099 | intel_vgpu_shadow_mm_unpin(workload); | ||
| 1100 | intel_vgpu_destroy_workload(workload); | ||
| 1101 | |||
| 1016 | atomic_dec(&s->running_workload_num); | 1102 | atomic_dec(&s->running_workload_num); |
| 1017 | wake_up(&scheduler->workload_complete_wq); | 1103 | wake_up(&scheduler->workload_complete_wq); |
| 1018 | 1104 | ||
| @@ -1406,6 +1492,16 @@ void intel_vgpu_destroy_workload(struct intel_vgpu_workload *workload) · 387 unmodified lines | |||
| 1406 | release_shadow_batch_buffer(workload); | 1492 | release_shadow_batch_buffer(workload); |
| 1407 | release_shadow_wa_ctx(&workload->wa_ctx); | 1493 | release_shadow_wa_ctx(&workload->wa_ctx); |
| 1408 | 1494 | ||
| 1495 | if (!list_empty(&workload->lri_shadow_mm)) { | ||
| 1496 | struct intel_vgpu_mm *m, *mm; | ||
| 1497 | list_for_each_entry_safe(m, mm, &workload->lri_shadow_mm, | ||
| 1498 | ppgtt_mm.link) { | ||
| 1499 | list_del(&m->ppgtt_mm.link); | ||
| 1500 | intel_vgpu_mm_put(m); | ||
| 1501 | } | ||
| 1502 | } | ||
| 1503 | |||
| 1504 | GEM_BUG_ON(!list_empty(&workload->lri_shadow_mm)); | ||
| 1409 | if (workload->shadow_mm) | 1505 | if (workload->shadow_mm) |
| 1410 | intel_vgpu_mm_put(workload->shadow_mm); | 1506 | intel_vgpu_mm_put(workload->shadow_mm); |
| 1411 | 1507 | ||
| @@ -1424,6 +1520,7 @@ alloc_workload(struct intel_vgpu *vgpu) · 12 unmodified lines | |||
| 1424 | 1520 | ||
| 1425 | INIT_LIST_HEAD(&workload->list); | 1521 | INIT_LIST_HEAD(&workload->list); |
| 1426 | INIT_LIST_HEAD(&workload->shadow_bb); | 1522 | INIT_LIST_HEAD(&workload->shadow_bb); |
| 1523 | INIT_LIST_HEAD(&workload->lri_shadow_mm); | ||
| 1427 | 1524 | ||
| 1428 | init_waitqueue_head(&workload->shadow_ctx_status_wq); | 1525 | init_waitqueue_head(&workload->shadow_ctx_status_wq); |
| 1429 | atomic_set(&workload->shadow_ctx_active, 0); | 1526 | atomic_set(&workload->shadow_ctx_active, 0); |
| @@ -87,6 +87,7 @@ struct intel_vgpu_workload { · 86 unmodified lines | |||
| 87 | int status; | 87 | int status; |
| 88 | 88 | ||
| 89 | struct intel_vgpu_mm *shadow_mm; | 89 | struct intel_vgpu_mm *shadow_mm; |
| 90 | struct list_head lri_shadow_mm; /* For PPGTT load cmd */ | ||
| 90 | 91 | ||
| 91 | /* different submission model may need different handler */ | 92 | /* different submission model may need different handler */ |
| 92 | int (*prepare)(struct intel_vgpu_workload *); | 93 | int (*prepare)(struct intel_vgpu_workload *); |
| @@ -0,0 +1,15 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright © 2020 Intel Corporation | ||
| 4 | */ | ||
| 5 | |||
| 6 | #include "i915_drv.h" | ||
| 7 | |||
| 8 | unsigned long | ||
| 9 | i915_fence_context_timeout(const struct drm_i915_private *i915, u64 context) | ||
| 10 | { | ||
| 11 | if (context && IS_ACTIVE(CONFIG_DRM_I915_FENCE_TIMEOUT)) | ||
| 12 | return msecs_to_jiffies_timeout(CONFIG_DRM_I915_FENCE_TIMEOUT); | ||
| 13 | |||
| 14 | return 0; | ||
| 15 | } | ||
| @@ -32,6 +32,7 @@ · 31 unmodified lines | |||
| 32 | #include <drm/drm_debugfs.h> | 32 | #include <drm/drm_debugfs.h> |
| 33 | 33 | ||
| 34 | #include "gem/i915_gem_context.h" | 34 | #include "gem/i915_gem_context.h" |
| 35 | #include "gt/intel_gt_buffer_pool.h" | ||
| 35 | #include "gt/intel_gt_clock_utils.h" | 36 | #include "gt/intel_gt_clock_utils.h" |
| 36 | #include "gt/intel_gt_pm.h" | 37 | #include "gt/intel_gt_pm.h" |
| 37 | #include "gt/intel_gt_requests.h" | 38 | #include "gt/intel_gt_requests.h" |
| @@ -1303,8 +1304,8 @@ static int i915_engine_info(struct seq_file *m, void *unused) · 1265 unmodified lines | |||
| 1303 | seq_printf(m, "GT awake? %s [%d]\n", | 1304 | seq_printf(m, "GT awake? %s [%d]\n", |
| 1304 | yesno(dev_priv->gt.awake), | 1305 | yesno(dev_priv->gt.awake), |
| 1305 | atomic_read(&dev_priv->gt.wakeref.count)); | 1306 | atomic_read(&dev_priv->gt.wakeref.count)); |
| 1306 | seq_printf(m, "CS timestamp frequency: %u kHz\n", | 1307 | seq_printf(m, "CS timestamp frequency: %u Hz\n", |
| 1307 | RUNTIME_INFO(dev_priv)->cs_timestamp_frequency_khz); | 1308 | RUNTIME_INFO(dev_priv)->cs_timestamp_frequency_hz); |
| 1308 | 1309 | ||
| 1309 | p = drm_seq_file_printer(m); | 1310 | p = drm_seq_file_printer(m); |
| 1310 | for_each_uabi_engine(engine, dev_priv) | 1311 | for_each_uabi_engine(engine, dev_priv) |
| @@ -1403,13 +1404,12 @@ static int · 92 unmodified lines | |||
| 1403 | i915_perf_noa_delay_set(void *data, u64 val) | 1404 | i915_perf_noa_delay_set(void *data, u64 val) |
| 1404 | { | 1405 | { |
| 1405 | struct drm_i915_private *i915 = data; | 1406 | struct drm_i915_private *i915 = data; |
| 1406 | const u32 clk = RUNTIME_INFO(i915)->cs_timestamp_frequency_khz; | ||
| 1407 | 1407 | ||
| 1408 | /* | 1408 | /* |
| 1409 | * This would lead to infinite waits as we're doing timestamp | 1409 | * This would lead to infinite waits as we're doing timestamp |
| 1410 | * difference on the CS with only 32bits. | 1410 | * difference on the CS with only 32bits. |
| 1411 | */ | 1411 | */ |
| 1412 | if (val > mul_u32_u32(U32_MAX, clk)) | 1412 | if (i915_cs_timestamp_ns_to_ticks(i915, val) > U32_MAX) |
| 1413 | return -EINVAL; | 1413 | return -EINVAL; |
| 1414 | 1414 | ||
| 1415 | atomic64_set(&i915->perf.noa_programming_delay, val); | 1415 | atomic64_set(&i915->perf.noa_programming_delay, val); |
| @@ -1484,6 +1484,9 @@ gt_drop_caches(struct intel_gt *gt, u64 val) · 68 unmodified lines | |||
| 1484 | if (val & DROP_RESET_ACTIVE && intel_gt_terminally_wedged(gt)) | 1484 | if (val & DROP_RESET_ACTIVE && intel_gt_terminally_wedged(gt)) |
| 1485 | intel_gt_handle_error(gt, ALL_ENGINES, 0, NULL); | 1485 | intel_gt_handle_error(gt, ALL_ENGINES, 0, NULL); |
| 1486 | 1486 | ||
| 1487 | if (val & DROP_FREED) | ||
| 1488 | intel_gt_flush_buffer_pool(gt); | ||
| 1489 | |||
| 1487 | return 0; | 1490 | return 0; |
| 1488 | } | 1491 | } |
| 1489 | 1492 | ||
| @@ -108,8 +108,8 @@ · 107 unmodified lines | |||
| 108 | 108 | ||
| 109 | #define DRIVER_NAME "i915" | 109 | #define DRIVER_NAME "i915" |
| 110 | #define DRIVER_DESC "Intel Graphics" | 110 | #define DRIVER_DESC "Intel Graphics" |
| 111 | #define DRIVER_DATE "20200430" | 111 | #define DRIVER_DATE "20200515" |
| 112 | #define DRIVER_TIMESTAMP 1588234401 | 112 | #define DRIVER_TIMESTAMP 1589543364 |
| 113 | 113 | ||
| 114 | struct drm_i915_gem_object; | 114 | struct drm_i915_gem_object; |
| 115 | 115 | ||
| @@ -148,6 +148,8 @@ enum hpd_pin { · 32 unmodified lines | |||
| 148 | struct i915_hotplug { | 148 | struct i915_hotplug { |
| 149 | struct delayed_work hotplug_work; | 149 | struct delayed_work hotplug_work; |
| 150 | 150 | ||
| 151 | const u32 *hpd, *pch_hpd; | ||
| 152 | |||
| 151 | struct { | 153 | struct { |
| 152 | unsigned long last_jiffies; | 154 | unsigned long last_jiffies; |
| 153 | int count; | 155 | int count; |
| @@ -510,6 +512,7 @@ struct i915_psr { · 356 unmodified lines | |||
| 510 | u32 dc3co_exit_delay; | 512 | u32 dc3co_exit_delay; |
| 511 | struct delayed_work dc3co_work; | 513 | struct delayed_work dc3co_work; |
| 512 | bool force_mode_changed; | 514 | bool force_mode_changed; |
| 515 | struct drm_dp_vsc_sdp vsc; | ||
| 513 | }; | 516 | }; |
| 514 | 517 | ||
| 515 | #define QUIRK_LVDS_SSC_DISABLE (1<<1) | 518 | #define QUIRK_LVDS_SSC_DISABLE (1<<1) |
| @@ -614,13 +617,14 @@ struct i915_gem_mm { · 98 unmodified lines | |||
| 614 | 617 | ||
| 615 | #define I915_IDLE_ENGINES_TIMEOUT (200) /* in ms */ | 618 | #define I915_IDLE_ENGINES_TIMEOUT (200) /* in ms */ |
| 616 | 619 | ||
| 617 | #define I915_RESET_TIMEOUT (10 * HZ) /* 10s */ | 620 | unsigned long i915_fence_context_timeout(const struct drm_i915_private *i915, |
| 618 | #define I915_FENCE_TIMEOUT (10 * HZ) /* 10s */ | 621 | u64 context); |
| 619 | |||
| 620 | #define I915_ENGINE_DEAD_TIMEOUT (4 * HZ) /* Seqno, head and subunits dead */ | ||
| 621 | #define I915_SEQNO_DEAD_TIMEOUT (12 * HZ) /* Seqno dead with active head */ | ||
| 622 | 622 | ||
| 623 | #define I915_ENGINE_WEDGED_TIMEOUT (60 * HZ) /* Reset but no recovery? */ | 623 | static inline unsigned long |
| 624 | i915_fence_timeout(const struct drm_i915_private *i915) | ||
| 625 | { | ||
| 626 | return i915_fence_context_timeout(i915, U64_MAX); | ||
| 627 | } | ||
| 624 | 628 | ||
| 625 | /* Amount of SAGV/QGV points, BSpec precisely defines this */ | 629 | /* Amount of SAGV/QGV points, BSpec precisely defines this */ |
| 626 | #define I915_NUM_QGV_POINTS 8 | 630 | #define I915_NUM_QGV_POINTS 8 |
| @@ -1507,6 +1511,11 @@ IS_SUBPLATFORM(const struct drm_i915_private *i915, · 880 unmodified lines | |||
| 1507 | #define IS_ICL_REVID(p, since, until) \ | 1511 | #define IS_ICL_REVID(p, since, until) \ |
| 1508 | (IS_ICELAKE(p) && IS_REVID(p, since, until)) | 1512 | (IS_ICELAKE(p) && IS_REVID(p, since, until)) |
| 1509 | 1513 | ||
| 1514 | #define EHL_REVID_A0 0x0 | ||
| 1515 | |||
| 1516 | #define IS_EHL_REVID(p, since, until) \ | ||
| 1517 | (IS_ELKHARTLAKE(p) && IS_REVID(p, since, until)) | ||
| 1518 | |||
| 1510 | #define TGL_REVID_A0 0x0 | 1519 | #define TGL_REVID_A0 0x0 |
| 1511 | #define TGL_REVID_B0 0x1 | 1520 | #define TGL_REVID_B0 0x1 |
| 1512 | #define TGL_REVID_C0 0x2 | 1521 | #define TGL_REVID_C0 0x2 |
| @@ -1915,4 +1924,16 @@ i915_coherent_map_type(struct drm_i915_private *i915) · 402 unmodified lines | |||
| 1915 | return HAS_LLC(i915) ? I915_MAP_WB : I915_MAP_WC; | 1924 | return HAS_LLC(i915) ? I915_MAP_WB : I915_MAP_WC; |
| 1916 | } | 1925 | } |
| 1917 | 1926 | ||
| 1927 | static inline u64 i915_cs_timestamp_ns_to_ticks(struct drm_i915_private *i915, u64 val) | ||
| 1928 | { | ||
| 1929 | return DIV_ROUND_UP_ULL(val * RUNTIME_INFO(i915)->cs_timestamp_frequency_hz, | ||
| 1930 | 1000000000); | ||
| 1931 | } | ||
| 1932 | |||
| 1933 | static inline u64 i915_cs_timestamp_ticks_to_ns(struct drm_i915_private *i915, u64 val) | ||
| 1934 | { | ||
| 1935 | return div_u64(val * 1000000000, | ||
| 1936 | RUNTIME_INFO(i915)->cs_timestamp_frequency_hz); | ||
| 1937 | } | ||
| 1938 | |||
| 1918 | #endif | 1939 | #endif |
| @@ -128,6 +128,13 @@ search_again: · 127 unmodified lines | |||
| 128 | active = NULL; | 128 | active = NULL; |
| 129 | INIT_LIST_HEAD(&eviction_list); | 129 | INIT_LIST_HEAD(&eviction_list); |
| 130 | list_for_each_entry_safe(vma, next, &vm->bound_list, vm_link) { | 130 | list_for_each_entry_safe(vma, next, &vm->bound_list, vm_link) { |
| 131 | if (vma == active) { /* now seen this vma twice */ | ||
| 132 | if (flags & PIN_NONBLOCK) | ||
| 133 | break; | ||
| 134 | |||
| 135 | active = ERR_PTR(-EAGAIN); | ||
| 136 | } | ||
| 137 | |||
| 131 | /* | 138 | /* |
| 132 | * We keep this list in a rough least-recently scanned order | 139 | * We keep this list in a rough least-recently scanned order |
| 133 | * of active elements (inactive elements are cheap to reap). | 140 | * of active elements (inactive elements are cheap to reap). |
| @@ -143,21 +150,12 @@ search_again: · 9 unmodified lines | |||
| 143 | * To notice when we complete one full cycle, we record the | 150 | * To notice when we complete one full cycle, we record the |
| 144 | * first active element seen, before moving it to the tail. | 151 | * first active element seen, before moving it to the tail. |
| 145 | */ | 152 | */ |
| 146 | if (i915_vma_is_active(vma)) { | 153 | if (active != ERR_PTR(-EAGAIN) && i915_vma_is_active(vma)) { |
| 147 | if (vma == active) { | 154 | if (!active) |
| 148 | if (flags & PIN_NONBLOCK) | 155 | active = vma; |
| 149 | break; | ||
| 150 | |||
| 151 | active = ERR_PTR(-EAGAIN); | ||
| 152 | } | ||
| 153 | |||
| 154 | if (active != ERR_PTR(-EAGAIN)) { | ||
| 155 | if (!active) | ||
| 156 | active = vma; | ||
| 157 | 156 | ||
| 158 | list_move_tail(&vma->vm_link, &vm->bound_list); | 157 | list_move_tail(&vma->vm_link, &vm->bound_list); |
| 159 | continue; | 158 | continue; |
| 160 | } | ||
| 161 | } | 159 | } |
| 162 | 160 | ||
| 163 | if (mark_free(&scan, vma, flags, &eviction_list)) | 161 | if (mark_free(&scan, vma, flags, &eviction_list)) |
| @@ -153,7 +153,7 @@ int i915_getparam_ioctl(struct drm_device *dev, void *data, · 152 unmodified lines | |||
| 153 | return -ENODEV; | 153 | return -ENODEV; |
| 154 | break; | 154 | break; |
| 155 | case I915_PARAM_CS_TIMESTAMP_FREQUENCY: | 155 | case I915_PARAM_CS_TIMESTAMP_FREQUENCY: |
| 156 | value = 1000 * RUNTIME_INFO(i915)->cs_timestamp_frequency_khz; | 156 | value = RUNTIME_INFO(i915)->cs_timestamp_frequency_hz; |
| 157 | break; | 157 | break; |
| 158 | case I915_PARAM_MMAP_GTT_COHERENT: | 158 | case I915_PARAM_MMAP_GTT_COHERENT: |
| 159 | value = INTEL_INFO(i915)->has_coherent_ggtt; | 159 | value = INTEL_INFO(i915)->has_coherent_ggtt; |
| @@ -42,7 +42,7 @@ struct i915_vma_coredump { · 41 unmodified lines | |||
| 42 | int num_pages; | 42 | int num_pages; |
| 43 | int page_count; | 43 | int page_count; |
| 44 | int unused; | 44 | int unused; |
| 45 | u32 *pages[0]; | 45 | u32 *pages[]; |
| 46 | }; | 46 | }; |
| 47 | 47 | ||
| 48 | struct i915_request_coredump { | 48 | struct i915_request_coredump { |
| @@ -124,7 +124,6 @@ static const u32 hpd_status_i915[HPD_NUM_PINS] = { · 123 unmodified lines | |||
| 124 | [HPD_PORT_D] = PORTD_HOTPLUG_INT_STATUS, | 124 | [HPD_PORT_D] = PORTD_HOTPLUG_INT_STATUS, |
| 125 | }; | 125 | }; |
| 126 | 126 | ||
| 127 | /* BXT hpd list */ | ||
| 128 | static const u32 hpd_bxt[HPD_NUM_PINS] = { | 127 | static const u32 hpd_bxt[HPD_NUM_PINS] = { |
| 129 | [HPD_PORT_A] = BXT_DE_PORT_HP_DDIA, | 128 | [HPD_PORT_A] = BXT_DE_PORT_HP_DDIA, |
| 130 | [HPD_PORT_B] = BXT_DE_PORT_HP_DDIB, | 129 | [HPD_PORT_B] = BXT_DE_PORT_HP_DDIB, |
| @@ -168,6 +167,49 @@ static const u32 hpd_tgp[HPD_NUM_PINS] = { · 37 unmodified lines | |||
| 168 | [HPD_PORT_I] = SDE_TC_HOTPLUG_ICP(PORT_TC6), | 167 | [HPD_PORT_I] = SDE_TC_HOTPLUG_ICP(PORT_TC6), |
| 169 | }; | 168 | }; |
| 170 | 169 | ||
| 170 | static void intel_hpd_init_pins(struct drm_i915_private *dev_priv) | ||
| 171 | { | ||
| 172 | struct i915_hotplug *hpd = &dev_priv->hotplug; | ||
| 173 | |||
| 174 | if (HAS_GMCH(dev_priv)) { | ||
| 175 | if (IS_G4X(dev_priv) || IS_VALLEYVIEW(dev_priv) || | ||
| 176 | IS_CHERRYVIEW(dev_priv)) | ||
| 177 | hpd->hpd = hpd_status_g4x; | ||
| 178 | else | ||
| 179 | hpd->hpd = hpd_status_i915; | ||
| 180 | return; | ||
| 181 | } | ||
| 182 | |||
| 183 | if (INTEL_GEN(dev_priv) >= 12) | ||
| 184 | hpd->hpd = hpd_gen12; | ||
| 185 | else if (INTEL_GEN(dev_priv) >= 11) | ||
| 186 | hpd->hpd = hpd_gen11; | ||
| 187 | else if (IS_GEN9_LP(dev_priv)) | ||
| 188 | hpd->hpd = hpd_bxt; | ||
| 189 | else if (INTEL_GEN(dev_priv) >= 8) | ||
| 190 | hpd->hpd = hpd_bdw; | ||
| 191 | else if (INTEL_GEN(dev_priv) >= 7) | ||
| 192 | hpd->hpd = hpd_ivb; | ||
| 193 | else | ||
| 194 | hpd->hpd = hpd_ilk; | ||
| 195 | |||
| 196 | if (!HAS_PCH_SPLIT(dev_priv) || HAS_PCH_NOP(dev_priv)) | ||
| 197 | return; | ||
| 198 | |||
| 199 | if (HAS_PCH_TGP(dev_priv) || HAS_PCH_JSP(dev_priv)) | ||
| 200 | hpd->pch_hpd = hpd_tgp; | ||
| 201 | else if (HAS_PCH_ICP(dev_priv) || HAS_PCH_MCC(dev_priv)) | ||
| 202 | hpd->pch_hpd = hpd_icp; | ||
| 203 | else if (HAS_PCH_CNP(dev_priv) || HAS_PCH_SPT(dev_priv)) | ||
| 204 | hpd->pch_hpd = hpd_spt; | ||
| 205 | else if (HAS_PCH_LPT(dev_priv) || HAS_PCH_CPT(dev_priv)) | ||
| 206 | hpd->pch_hpd = hpd_cpt; | ||
| 207 | else if (HAS_PCH_IBX(dev_priv)) | ||
| 208 | hpd->pch_hpd = hpd_ibx; | ||
| 209 | else | ||
| 210 | MISSING_CASE(INTEL_PCH_TYPE(dev_priv)); | ||
| 211 | } | ||
| 212 | |||
| 171 | static void | 213 | static void |
| 172 | intel_handle_vblank(struct drm_i915_private *dev_priv, enum pipe pipe) | 214 | intel_handle_vblank(struct drm_i915_private *dev_priv, enum pipe pipe) |
| 173 | { | 215 | { |
| @@ -1504,33 +1546,27 @@ static void i9xx_hpd_irq_handler(struct drm_i915_private *dev_priv, · 1330 unmodified lines | |||
| 1504 | u32 hotplug_status) | 1546 | u32 hotplug_status) |
| 1505 | { | 1547 | { |
| 1506 | u32 pin_mask = 0, long_mask = 0; | 1548 | u32 pin_mask = 0, long_mask = 0; |
| 1549 | u32 hotplug_trigger; | ||
| 1507 | 1550 | ||
| 1508 | if (IS_G4X(dev_priv) || IS_VALLEYVIEW(dev_priv) || | 1551 | if (IS_G4X(dev_priv) || |
| 1509 | IS_CHERRYVIEW(dev_priv)) { | 1552 | IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) |
| 1510 | u32 hotplug_trigger = hotplug_status & HOTPLUG_INT_STATUS_G4X; | 1553 | hotplug_trigger = hotplug_status & HOTPLUG_INT_STATUS_G4X; |
| 1511 | 1554 | else | |
| 1512 | if (hotplug_trigger) { | 1555 | hotplug_trigger = hotplug_status & HOTPLUG_INT_STATUS_I915; |
| 1513 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | ||
| 1514 | hotplug_trigger, hotplug_trigger, | ||
| 1515 | hpd_status_g4x, | ||
| 1516 | i9xx_port_hotplug_long_detect); | ||
| 1517 | 1556 | ||
| 1518 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); | 1557 | if (hotplug_trigger) { |
| 1519 | } | 1558 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1559 | hotplug_trigger, hotplug_trigger, | ||
| 1560 | dev_priv->hotplug.hpd, | ||
| 1561 | i9xx_port_hotplug_long_detect); | ||
| 1520 | 1562 | ||
| 1521 | if (hotplug_status & DP_AUX_CHANNEL_MASK_INT_STATUS_G4X) | 1563 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); |
| 1522 | dp_aux_irq_handler(dev_priv); | ||
| 1523 | } else { | ||
| 1524 | u32 hotplug_trigger = hotplug_status & HOTPLUG_INT_STATUS_I915; | ||
| 1525 | |||
| 1526 | if (hotplug_trigger) { | ||
| 1527 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | ||
| 1528 | hotplug_trigger, hotplug_trigger, | ||
| 1529 | hpd_status_i915, | ||
| 1530 | i9xx_port_hotplug_long_detect); | ||
| 1531 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); | ||
| 1532 | } | ||
| 1533 | } | 1564 | } |
| 1565 | |||
| 1566 | if ((IS_G4X(dev_priv) || | ||
| 1567 | IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) && | ||
| 1568 | hotplug_status & DP_AUX_CHANNEL_MASK_INT_STATUS_G4X) | ||
| 1569 | dp_aux_irq_handler(dev_priv); | ||
| 1534 | } | 1570 | } |
| 1535 | 1571 | ||
| 1536 | static irqreturn_t valleyview_irq_handler(int irq, void *arg) | 1572 | static irqreturn_t valleyview_irq_handler(int irq, void *arg) |
| @@ -1696,8 +1732,7 @@ static irqreturn_t cherryview_irq_handler(int irq, void *arg) · 159 unmodified lines | |||
| 1696 | } | 1732 | } |
| 1697 | 1733 | ||
| 1698 | static void ibx_hpd_irq_handler(struct drm_i915_private *dev_priv, | 1734 | static void ibx_hpd_irq_handler(struct drm_i915_private *dev_priv, |
| 1699 | u32 hotplug_trigger, | 1735 | u32 hotplug_trigger) |
| 1700 | const u32 hpd[HPD_NUM_PINS]) | ||
| 1701 | { | 1736 | { |
| 1702 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; | 1737 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; |
| 1703 | 1738 | ||
| @@ -1720,8 +1755,9 @@ static void ibx_hpd_irq_handler(struct drm_i915_private *dev_priv, · 16 unmodified lines | |||
| 1720 | if (!hotplug_trigger) | 1755 | if (!hotplug_trigger) |
| 1721 | return; | 1756 | return; |
| 1722 | 1757 | ||
| 1723 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, | 1758 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1724 | dig_hotplug_reg, hpd, | 1759 | hotplug_trigger, dig_hotplug_reg, |
| 1760 | dev_priv->hotplug.pch_hpd, | ||
| 1725 | pch_port_hotplug_long_detect); | 1761 | pch_port_hotplug_long_detect); |
| 1726 | 1762 | ||
| 1727 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); | 1763 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); |
| @@ -1732,7 +1768,7 @@ static void ibx_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 4 unmodified lines | |||
| 1732 | enum pipe pipe; | 1768 | enum pipe pipe; |
| 1733 | u32 hotplug_trigger = pch_iir & SDE_HOTPLUG_MASK; | 1769 | u32 hotplug_trigger = pch_iir & SDE_HOTPLUG_MASK; |
| 1734 | 1770 | ||
| 1735 | ibx_hpd_irq_handler(dev_priv, hotplug_trigger, hpd_ibx); | 1771 | ibx_hpd_irq_handler(dev_priv, hotplug_trigger); |
| 1736 | 1772 | ||
| 1737 | if (pch_iir & SDE_AUDIO_POWER_MASK) { | 1773 | if (pch_iir & SDE_AUDIO_POWER_MASK) { |
| 1738 | int port = ffs((pch_iir & SDE_AUDIO_POWER_MASK) >> | 1774 | int port = ffs((pch_iir & SDE_AUDIO_POWER_MASK) >> |
| @@ -1820,7 +1856,7 @@ static void cpt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 81 unmodified lines | |||
| 1820 | enum pipe pipe; | 1856 | enum pipe pipe; |
| 1821 | u32 hotplug_trigger = pch_iir & SDE_HOTPLUG_MASK_CPT; | 1857 | u32 hotplug_trigger = pch_iir & SDE_HOTPLUG_MASK_CPT; |
| 1822 | 1858 | ||
| 1823 | ibx_hpd_irq_handler(dev_priv, hotplug_trigger, hpd_cpt); | 1859 | ibx_hpd_irq_handler(dev_priv, hotplug_trigger); |
| 1824 | 1860 | ||
| 1825 | if (pch_iir & SDE_AUDIO_POWER_MASK_CPT) { | 1861 | if (pch_iir & SDE_AUDIO_POWER_MASK_CPT) { |
| 1826 | int port = ffs((pch_iir & SDE_AUDIO_POWER_MASK_CPT) >> | 1862 | int port = ffs((pch_iir & SDE_AUDIO_POWER_MASK_CPT) >> |
| @@ -1857,22 +1893,18 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 30 unmodified lines | |||
| 1857 | u32 ddi_hotplug_trigger, tc_hotplug_trigger; | 1893 | u32 ddi_hotplug_trigger, tc_hotplug_trigger; |
| 1858 | u32 pin_mask = 0, long_mask = 0; | 1894 | u32 pin_mask = 0, long_mask = 0; |
| 1859 | bool (*tc_port_hotplug_long_detect)(enum hpd_pin pin, u32 val); | 1895 | bool (*tc_port_hotplug_long_detect)(enum hpd_pin pin, u32 val); |
| 1860 | const u32 *pins; | ||
| 1861 | 1896 | ||
| 1862 | if (HAS_PCH_TGP(dev_priv)) { | 1897 | if (HAS_PCH_TGP(dev_priv)) { |
| 1863 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_TGP; | 1898 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_TGP; |
| 1864 | tc_hotplug_trigger = pch_iir & SDE_TC_MASK_TGP; | 1899 | tc_hotplug_trigger = pch_iir & SDE_TC_MASK_TGP; |
| 1865 | tc_port_hotplug_long_detect = tgp_tc_port_hotplug_long_detect; | 1900 | tc_port_hotplug_long_detect = tgp_tc_port_hotplug_long_detect; |
| 1866 | pins = hpd_tgp; | ||
| 1867 | } else if (HAS_PCH_JSP(dev_priv)) { | 1901 | } else if (HAS_PCH_JSP(dev_priv)) { |
| 1868 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_TGP; | 1902 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_TGP; |
| 1869 | tc_hotplug_trigger = 0; | 1903 | tc_hotplug_trigger = 0; |
| 1870 | pins = hpd_tgp; | ||
| 1871 | } else if (HAS_PCH_MCC(dev_priv)) { | 1904 | } else if (HAS_PCH_MCC(dev_priv)) { |
| 1872 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_ICP; | 1905 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_ICP; |
| 1873 | tc_hotplug_trigger = pch_iir & SDE_TC_HOTPLUG_ICP(PORT_TC1); | 1906 | tc_hotplug_trigger = pch_iir & SDE_TC_HOTPLUG_ICP(PORT_TC1); |
| 1874 | tc_port_hotplug_long_detect = icp_tc_port_hotplug_long_detect; | 1907 | tc_port_hotplug_long_detect = icp_tc_port_hotplug_long_detect; |
| 1875 | pins = hpd_icp; | ||
| 1876 | } else { | 1908 | } else { |
| 1877 | drm_WARN(&dev_priv->drm, !HAS_PCH_ICP(dev_priv), | 1909 | drm_WARN(&dev_priv->drm, !HAS_PCH_ICP(dev_priv), |
| 1878 | "Unrecognized PCH type 0x%x\n", | 1910 | "Unrecognized PCH type 0x%x\n", |
| @@ -1881,7 +1913,6 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 2 unmodified lines | |||
| 1881 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_ICP; | 1913 | ddi_hotplug_trigger = pch_iir & SDE_DDI_MASK_ICP; |
| 1882 | tc_hotplug_trigger = pch_iir & SDE_TC_MASK_ICP; | 1914 | tc_hotplug_trigger = pch_iir & SDE_TC_MASK_ICP; |
| 1883 | tc_port_hotplug_long_detect = icp_tc_port_hotplug_long_detect; | 1915 | tc_port_hotplug_long_detect = icp_tc_port_hotplug_long_detect; |
| 1884 | pins = hpd_icp; | ||
| 1885 | } | 1916 | } |
| 1886 | 1917 | ||
| 1887 | if (ddi_hotplug_trigger) { | 1918 | if (ddi_hotplug_trigger) { |
| @@ -1891,8 +1922,8 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 3 unmodified lines | |||
| 1891 | I915_WRITE(SHOTPLUG_CTL_DDI, dig_hotplug_reg); | 1922 | I915_WRITE(SHOTPLUG_CTL_DDI, dig_hotplug_reg); |
| 1892 | 1923 | ||
| 1893 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | 1924 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1894 | ddi_hotplug_trigger, | 1925 | ddi_hotplug_trigger, dig_hotplug_reg, |
| 1895 | dig_hotplug_reg, pins, | 1926 | dev_priv->hotplug.pch_hpd, |
| 1896 | icp_ddi_port_hotplug_long_detect); | 1927 | icp_ddi_port_hotplug_long_detect); |
| 1897 | } | 1928 | } |
| 1898 | 1929 | ||
| @@ -1903,8 +1934,8 @@ static void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 4 unmodified lines | |||
| 1903 | I915_WRITE(SHOTPLUG_CTL_TC, dig_hotplug_reg); | 1934 | I915_WRITE(SHOTPLUG_CTL_TC, dig_hotplug_reg); |
| 1904 | 1935 | ||
| 1905 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | 1936 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1906 | tc_hotplug_trigger, | 1937 | tc_hotplug_trigger, dig_hotplug_reg, |
| 1907 | dig_hotplug_reg, pins, | 1938 | dev_priv->hotplug.pch_hpd, |
| 1908 | tc_port_hotplug_long_detect); | 1939 | tc_port_hotplug_long_detect); |
| 1909 | } | 1940 | } |
| 1910 | 1941 | ||
| @@ -1929,7 +1960,8 @@ static void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 18 unmodified lines | |||
| 1929 | I915_WRITE(PCH_PORT_HOTPLUG, dig_hotplug_reg); | 1960 | I915_WRITE(PCH_PORT_HOTPLUG, dig_hotplug_reg); |
| 1930 | 1961 | ||
| 1931 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | 1962 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1932 | hotplug_trigger, dig_hotplug_reg, hpd_spt, | 1963 | hotplug_trigger, dig_hotplug_reg, |
| 1964 | dev_priv->hotplug.pch_hpd, | ||
| 1933 | spt_port_hotplug_long_detect); | 1965 | spt_port_hotplug_long_detect); |
| 1934 | } | 1966 | } |
| 1935 | 1967 | ||
| @@ -1940,7 +1972,8 @@ static void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 4 unmodified lines | |||
| 1940 | I915_WRITE(PCH_PORT_HOTPLUG2, dig_hotplug_reg); | 1972 | I915_WRITE(PCH_PORT_HOTPLUG2, dig_hotplug_reg); |
| 1941 | 1973 | ||
| 1942 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, | 1974 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1943 | hotplug2_trigger, dig_hotplug_reg, hpd_spt, | 1975 | hotplug2_trigger, dig_hotplug_reg, |
| 1976 | dev_priv->hotplug.pch_hpd, | ||
| 1944 | spt_port_hotplug2_long_detect); | 1977 | spt_port_hotplug2_long_detect); |
| 1945 | } | 1978 | } |
| 1946 | 1979 | ||
| @@ -1952,16 +1985,16 @@ static void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir) · 5 unmodified lines | |||
| 1952 | } | 1985 | } |
| 1953 | 1986 | ||
| 1954 | static void ilk_hpd_irq_handler(struct drm_i915_private *dev_priv, | 1987 | static void ilk_hpd_irq_handler(struct drm_i915_private *dev_priv, |
| 1955 | u32 hotplug_trigger, | 1988 | u32 hotplug_trigger) |
| 1956 | const u32 hpd[HPD_NUM_PINS]) | ||
| 1957 | { | 1989 | { |
| 1958 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; | 1990 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; |
| 1959 | 1991 | ||
| 1960 | dig_hotplug_reg = I915_READ(DIGITAL_PORT_HOTPLUG_CNTRL); | 1992 | dig_hotplug_reg = I915_READ(DIGITAL_PORT_HOTPLUG_CNTRL); |
| 1961 | I915_WRITE(DIGITAL_PORT_HOTPLUG_CNTRL, dig_hotplug_reg); | 1993 | I915_WRITE(DIGITAL_PORT_HOTPLUG_CNTRL, dig_hotplug_reg); |
| 1962 | 1994 | ||
| 1963 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, | 1995 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 1964 | dig_hotplug_reg, hpd, | 1996 | hotplug_trigger, dig_hotplug_reg, |
| 1997 | dev_priv->hotplug.hpd, | ||
| 1965 | ilk_port_hotplug_long_detect); | 1998 | ilk_port_hotplug_long_detect); |
| 1966 | 1999 | ||
| 1967 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); | 2000 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); |
| @@ -1974,7 +2007,7 @@ static void ilk_display_irq_handler(struct drm_i915_private *dev_priv, · 6 unmodified lines | |||
| 1974 | u32 hotplug_trigger = de_iir & DE_DP_A_HOTPLUG; | 2007 | u32 hotplug_trigger = de_iir & DE_DP_A_HOTPLUG; |
| 1975 | 2008 | ||
| 1976 | if (hotplug_trigger) | 2009 | if (hotplug_trigger) |
| 1977 | ilk_hpd_irq_handler(dev_priv, hotplug_trigger, hpd_ilk); | 2010 | ilk_hpd_irq_handler(dev_priv, hotplug_trigger); |
| 1978 | 2011 | ||
| 1979 | if (de_iir & DE_AUX_CHANNEL_A) | 2012 | if (de_iir & DE_AUX_CHANNEL_A) |
| 1980 | dp_aux_irq_handler(dev_priv); | 2013 | dp_aux_irq_handler(dev_priv); |
| @@ -2020,7 +2053,7 @@ static void ivb_display_irq_handler(struct drm_i915_private *dev_priv, · 39 unmodified lines | |||
| 2020 | u32 hotplug_trigger = de_iir & DE_DP_A_HOTPLUG_IVB; | 2053 | u32 hotplug_trigger = de_iir & DE_DP_A_HOTPLUG_IVB; |
| 2021 | 2054 | ||
| 2022 | if (hotplug_trigger) | 2055 | if (hotplug_trigger) |
| 2023 | ilk_hpd_irq_handler(dev_priv, hotplug_trigger, hpd_ivb); | 2056 | ilk_hpd_irq_handler(dev_priv, hotplug_trigger); |
| 2024 | 2057 | ||
| 2025 | if (de_iir & DE_ERR_INT_IVB) | 2058 | if (de_iir & DE_ERR_INT_IVB) |
| 2026 | ivb_err_int_handler(dev_priv); | 2059 | ivb_err_int_handler(dev_priv); |
| @@ -2130,16 +2163,16 @@ static irqreturn_t ilk_irq_handler(int irq, void *arg) · 103 unmodified lines | |||
| 2130 | } | 2163 | } |
| 2131 | 2164 | ||
| 2132 | static void bxt_hpd_irq_handler(struct drm_i915_private *dev_priv, | 2165 | static void bxt_hpd_irq_handler(struct drm_i915_private *dev_priv, |
| 2133 | u32 hotplug_trigger, | 2166 | u32 hotplug_trigger) |
| 2134 | const u32 hpd[HPD_NUM_PINS]) | ||
| 2135 | { | 2167 | { |
| 2136 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; | 2168 | u32 dig_hotplug_reg, pin_mask = 0, long_mask = 0; |
| 2137 | 2169 | ||
| 2138 | dig_hotplug_reg = I915_READ(PCH_PORT_HOTPLUG); | 2170 | dig_hotplug_reg = I915_READ(PCH_PORT_HOTPLUG); |
| 2139 | I915_WRITE(PCH_PORT_HOTPLUG, dig_hotplug_reg); | 2171 | I915_WRITE(PCH_PORT_HOTPLUG, dig_hotplug_reg); |
| 2140 | 2172 | ||
| 2141 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, hotplug_trigger, | 2173 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 2142 | dig_hotplug_reg, hpd, | 2174 | hotplug_trigger, dig_hotplug_reg, |
| 2175 | dev_priv->hotplug.hpd, | ||
| 2143 | bxt_port_hotplug_long_detect); | 2176 | bxt_port_hotplug_long_detect); |
| 2144 | 2177 | ||
| 2145 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); | 2178 | intel_hpd_irq_handler(dev_priv, pin_mask, long_mask); |
| @@ -2151,15 +2184,11 @@ static void gen11_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 iir) · 5 unmodified lines | |||
| 2151 | u32 trigger_tc = iir & GEN11_DE_TC_HOTPLUG_MASK; | 2184 | u32 trigger_tc = iir & GEN11_DE_TC_HOTPLUG_MASK; |
| 2152 | u32 trigger_tbt = iir & GEN11_DE_TBT_HOTPLUG_MASK; | 2185 | u32 trigger_tbt = iir & GEN11_DE_TBT_HOTPLUG_MASK; |
| 2153 | long_pulse_detect_func long_pulse_detect; | 2186 | long_pulse_detect_func long_pulse_detect; |
| 2154 | const u32 *hpd; | ||
| 2155 | 2187 | ||
| 2156 | if (INTEL_GEN(dev_priv) >= 12) { | 2188 | if (INTEL_GEN(dev_priv) >= 12) |
| 2157 | long_pulse_detect = gen12_port_hotplug_long_detect; | 2189 | long_pulse_detect = gen12_port_hotplug_long_detect; |
| 2158 | hpd = hpd_gen12; | 2190 | else |
| 2159 | } else { | ||
| 2160 | long_pulse_detect = gen11_port_hotplug_long_detect; | 2191 | long_pulse_detect = gen11_port_hotplug_long_detect; |
| 2161 | hpd = hpd_gen11; | ||
| 2162 | } | ||
| 2163 | 2192 | ||
| 2164 | if (trigger_tc) { | 2193 | if (trigger_tc) { |
| 2165 | u32 dig_hotplug_reg; | 2194 | u32 dig_hotplug_reg; |
| @@ -2167,8 +2196,10 @@ static void gen11_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 iir) · 1 unmodified line | |||
| 2167 | dig_hotplug_reg = I915_READ(GEN11_TC_HOTPLUG_CTL); | 2196 | dig_hotplug_reg = I915_READ(GEN11_TC_HOTPLUG_CTL); |
| 2168 | I915_WRITE(GEN11_TC_HOTPLUG_CTL, dig_hotplug_reg); | 2197 | I915_WRITE(GEN11_TC_HOTPLUG_CTL, dig_hotplug_reg); |
| 2169 | 2198 | ||
| 2170 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, trigger_tc, | 2199 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 2171 | dig_hotplug_reg, hpd, long_pulse_detect); | 2200 | trigger_tc, dig_hotplug_reg, |
| 2201 | dev_priv->hotplug.hpd, | ||
| 2202 | long_pulse_detect); | ||
| 2172 | } | 2203 | } |
| 2173 | 2204 | ||
| 2174 | if (trigger_tbt) { | 2205 | if (trigger_tbt) { |
| @@ -2177,8 +2208,10 @@ static void gen11_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 iir) · 2 unmodified lines | |||
| 2177 | dig_hotplug_reg = I915_READ(GEN11_TBT_HOTPLUG_CTL); | 2208 | dig_hotplug_reg = I915_READ(GEN11_TBT_HOTPLUG_CTL); |
| 2178 | I915_WRITE(GEN11_TBT_HOTPLUG_CTL, dig_hotplug_reg); | 2209 | I915_WRITE(GEN11_TBT_HOTPLUG_CTL, dig_hotplug_reg); |
| 2179 | 2210 | ||
| 2180 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, trigger_tbt, | 2211 | intel_get_hpd_pins(dev_priv, &pin_mask, &long_mask, |
| 2181 | dig_hotplug_reg, hpd, long_pulse_detect); | 2212 | trigger_tbt, dig_hotplug_reg, |
| 2213 | dev_priv->hotplug.hpd, | ||
| 2214 | long_pulse_detect); | ||
| 2182 | } | 2215 | } |
| 2183 | 2216 | ||
| 2184 | if (pin_mask) | 2217 | if (pin_mask) |
| @@ -2309,15 +2342,13 @@ gen8_de_irq_handler(struct drm_i915_private *dev_priv, u32 master_ctl) · 124 unmodified lines | |||
| 2309 | if (IS_GEN9_LP(dev_priv)) { | 2342 | if (IS_GEN9_LP(dev_priv)) { |
| 2310 | tmp_mask = iir & BXT_DE_PORT_HOTPLUG_MASK; | 2343 | tmp_mask = iir & BXT_DE_PORT_HOTPLUG_MASK; |
| 2311 | if (tmp_mask) { | 2344 | if (tmp_mask) { |
| 2312 | bxt_hpd_irq_handler(dev_priv, tmp_mask, | 2345 | bxt_hpd_irq_handler(dev_priv, tmp_mask); |
| 2313 | hpd_bxt); | ||
| 2314 | found = true; | 2346 | found = true; |
| 2315 | } | 2347 | } |
| 2316 | } else if (IS_BROADWELL(dev_priv)) { | 2348 | } else if (IS_BROADWELL(dev_priv)) { |
| 2317 | tmp_mask = iir & GEN8_PORT_DP_A_HOTPLUG; | 2349 | tmp_mask = iir & GEN8_PORT_DP_A_HOTPLUG; |
| 2318 | if (tmp_mask) { | 2350 | if (tmp_mask) { |
| 2319 | ilk_hpd_irq_handler(dev_priv, | 2351 | ilk_hpd_irq_handler(dev_priv, tmp_mask); |
| 2320 | tmp_mask, hpd_bdw); | ||
| 2321 | found = true; | 2352 | found = true; |
| 2322 | } | 2353 | } |
| 2323 | } | 2354 | } |
| @@ -2870,6 +2901,14 @@ static void gen11_display_irq_reset(struct drm_i915_private *dev_priv) · 546 unmodified lines | |||
| 2870 | 2901 | ||
| 2871 | if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP) | 2902 | if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP) |
| 2872 | GEN3_IRQ_RESET(uncore, SDE); | 2903 | GEN3_IRQ_RESET(uncore, SDE); |
| 2904 | |||
| 2905 | /* Wa_14010685332:icl */ | ||
| 2906 | if (INTEL_PCH_TYPE(dev_priv) == PCH_ICP) { | ||
| 2907 | intel_uncore_rmw(uncore, SOUTH_CHICKEN1, | ||
| 2908 | SBCLK_RUN_REFCLK_DIS, SBCLK_RUN_REFCLK_DIS); | ||
| 2909 | intel_uncore_rmw(uncore, SOUTH_CHICKEN1, | ||
| 2910 | SBCLK_RUN_REFCLK_DIS, 0); | ||
| 2911 | } | ||
| 2873 | } | 2912 | } |
| 2874 | 2913 | ||
| 2875 | static void gen11_irq_reset(struct drm_i915_private *dev_priv) | 2914 | static void gen11_irq_reset(struct drm_i915_private *dev_priv) |
| @@ -2989,13 +3028,12 @@ static void ibx_hpd_irq_setup(struct drm_i915_private *dev_priv) · 113 unmodified lines | |||
| 2989 | { | 3028 | { |
| 2990 | u32 hotplug_irqs, enabled_irqs; | 3029 | u32 hotplug_irqs, enabled_irqs; |
| 2991 | 3030 | ||
| 2992 | if (HAS_PCH_IBX(dev_priv)) { | 3031 | if (HAS_PCH_IBX(dev_priv)) |
| 2993 | hotplug_irqs = SDE_HOTPLUG_MASK; | 3032 | hotplug_irqs = SDE_HOTPLUG_MASK; |
| 2994 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_ibx); | 3033 | else |
| 2995 | } else { | ||
| 2996 | hotplug_irqs = SDE_HOTPLUG_MASK_CPT; | 3034 | hotplug_irqs = SDE_HOTPLUG_MASK_CPT; |
| 2997 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_cpt); | 3035 | |
| 2998 | } | 3036 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.pch_hpd); |
| 2999 | 3037 | ||
| 3000 | ibx_display_interrupt_update(dev_priv, hotplug_irqs, enabled_irqs); | 3038 | ibx_display_interrupt_update(dev_priv, hotplug_irqs, enabled_irqs); |
| 3001 | 3039 | ||
| @@ -3021,13 +3059,12 @@ static void icp_hpd_detection_setup(struct drm_i915_private *dev_priv, · 19 unmodified lines | |||
| 3021 | 3059 | ||
| 3022 | static void icp_hpd_irq_setup(struct drm_i915_private *dev_priv, | 3060 | static void icp_hpd_irq_setup(struct drm_i915_private *dev_priv, |
| 3023 | u32 sde_ddi_mask, u32 sde_tc_mask, | 3061 | u32 sde_ddi_mask, u32 sde_tc_mask, |
| 3024 | u32 ddi_enable_mask, u32 tc_enable_mask, | 3062 | u32 ddi_enable_mask, u32 tc_enable_mask) |
| 3025 | const u32 *pins) | ||
| 3026 | { | 3063 | { |
| 3027 | u32 hotplug_irqs, enabled_irqs; | 3064 | u32 hotplug_irqs, enabled_irqs; |
| 3028 | 3065 | ||
| 3029 | hotplug_irqs = sde_ddi_mask | sde_tc_mask; | 3066 | hotplug_irqs = sde_ddi_mask | sde_tc_mask; |
| 3030 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, pins); | 3067 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.pch_hpd); |
| 3031 | 3068 | ||
| 3032 | I915_WRITE(SHPD_FILTER_CNT, SHPD_FILTER_CNT_500_ADJ); | 3069 | I915_WRITE(SHPD_FILTER_CNT, SHPD_FILTER_CNT_500_ADJ); |
| 3033 | 3070 | ||
| @@ -3044,8 +3081,7 @@ static void mcc_hpd_irq_setup(struct drm_i915_private *dev_priv) · 10 unmodified lines | |||
| 3044 | { | 3081 | { |
| 3045 | icp_hpd_irq_setup(dev_priv, | 3082 | icp_hpd_irq_setup(dev_priv, |
| 3046 | SDE_DDI_MASK_ICP, SDE_TC_HOTPLUG_ICP(PORT_TC1), | 3083 | SDE_DDI_MASK_ICP, SDE_TC_HOTPLUG_ICP(PORT_TC1), |
| 3047 | ICP_DDI_HPD_ENABLE_MASK, ICP_TC_HPD_ENABLE(PORT_TC1), | 3084 | ICP_DDI_HPD_ENABLE_MASK, ICP_TC_HPD_ENABLE(PORT_TC1)); |
| 3048 | hpd_icp); | ||
| 3049 | } | 3085 | } |
| 3050 | 3086 | ||
| 3051 | /* | 3087 | /* |
| @@ -3057,8 +3093,7 @@ static void jsp_hpd_irq_setup(struct drm_i915_private *dev_priv) · 5 unmodified lines | |||
| 3057 | { | 3093 | { |
| 3058 | icp_hpd_irq_setup(dev_priv, | 3094 | icp_hpd_irq_setup(dev_priv, |
| 3059 | SDE_DDI_MASK_TGP, 0, | 3095 | SDE_DDI_MASK_TGP, 0, |
| 3060 | TGP_DDI_HPD_ENABLE_MASK, 0, | 3096 | TGP_DDI_HPD_ENABLE_MASK, 0); |
| 3061 | hpd_tgp); | ||
| 3062 | } | 3097 | } |
| 3063 | 3098 | ||
| 3064 | static void gen11_hpd_detection_setup(struct drm_i915_private *dev_priv) | 3099 | static void gen11_hpd_detection_setup(struct drm_i915_private *dev_priv) |
| @@ -3083,11 +3118,9 @@ static void gen11_hpd_detection_setup(struct drm_i915_private *dev_priv) · 18 unmodified lines | |||
| 3083 | static void gen11_hpd_irq_setup(struct drm_i915_private *dev_priv) | 3118 | static void gen11_hpd_irq_setup(struct drm_i915_private *dev_priv) |
| 3084 | { | 3119 | { |
| 3085 | u32 hotplug_irqs, enabled_irqs; | 3120 | u32 hotplug_irqs, enabled_irqs; |
| 3086 | const u32 *hpd; | ||
| 3087 | u32 val; | 3121 | u32 val; |
| 3088 | 3122 | ||
| 3089 | hpd = INTEL_GEN(dev_priv) >= 12 ? hpd_gen12 : hpd_gen11; | 3123 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.hpd); |
| 3090 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd); | ||
| 3091 | hotplug_irqs = GEN11_DE_TC_HOTPLUG_MASK | GEN11_DE_TBT_HOTPLUG_MASK; | 3124 | hotplug_irqs = GEN11_DE_TC_HOTPLUG_MASK | GEN11_DE_TBT_HOTPLUG_MASK; |
| 3092 | 3125 | ||
| 3093 | val = I915_READ(GEN11_DE_HPD_IMR); | 3126 | val = I915_READ(GEN11_DE_HPD_IMR); |
| @@ -3099,12 +3132,10 @@ static void gen11_hpd_irq_setup(struct drm_i915_private *dev_priv) · 5 unmodified lines | |||
| 3099 | 3132 | ||
| 3100 | if (INTEL_PCH_TYPE(dev_priv) >= PCH_TGP) | 3133 | if (INTEL_PCH_TYPE(dev_priv) >= PCH_TGP) |
| 3101 | icp_hpd_irq_setup(dev_priv, SDE_DDI_MASK_TGP, SDE_TC_MASK_TGP, | 3134 | icp_hpd_irq_setup(dev_priv, SDE_DDI_MASK_TGP, SDE_TC_MASK_TGP, |
| 3102 | TGP_DDI_HPD_ENABLE_MASK, | 3135 | TGP_DDI_HPD_ENABLE_MASK, TGP_TC_HPD_ENABLE_MASK); |
| 3103 | TGP_TC_HPD_ENABLE_MASK, hpd_tgp); | ||
| 3104 | else if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP) | 3136 | else if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP) |
| 3105 | icp_hpd_irq_setup(dev_priv, SDE_DDI_MASK_ICP, SDE_TC_MASK_ICP, | 3137 | icp_hpd_irq_setup(dev_priv, SDE_DDI_MASK_ICP, SDE_TC_MASK_ICP, |
| 3106 | ICP_DDI_HPD_ENABLE_MASK, | 3138 | ICP_DDI_HPD_ENABLE_MASK, ICP_TC_HPD_ENABLE_MASK); |
| 3107 | ICP_TC_HPD_ENABLE_MASK, hpd_icp); | ||
| 3108 | } | 3139 | } |
| 3109 | 3140 | ||
| 3110 | static void spt_hpd_detection_setup(struct drm_i915_private *dev_priv) | 3141 | static void spt_hpd_detection_setup(struct drm_i915_private *dev_priv) |
| @@ -3140,7 +3171,7 @@ static void spt_hpd_irq_setup(struct drm_i915_private *dev_priv) · 29 unmodified lines | |||
| 3140 | I915_WRITE(SHPD_FILTER_CNT, SHPD_FILTER_CNT_500_ADJ); | 3171 | I915_WRITE(SHPD_FILTER_CNT, SHPD_FILTER_CNT_500_ADJ); |
| 3141 | 3172 | ||
| 3142 | hotplug_irqs = SDE_HOTPLUG_MASK_SPT; | 3173 | hotplug_irqs = SDE_HOTPLUG_MASK_SPT; |
| 3143 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_spt); | 3174 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.pch_hpd); |
| 3144 | 3175 | ||
| 3145 | ibx_display_interrupt_update(dev_priv, hotplug_irqs, enabled_irqs); | 3176 | ibx_display_interrupt_update(dev_priv, hotplug_irqs, enabled_irqs); |
| 3146 | 3177 | ||
| @@ -3169,17 +3200,17 @@ static void ilk_hpd_irq_setup(struct drm_i915_private *dev_priv) · 22 unmodified lines | |||
| 3169 | 3200 | ||
| 3170 | if (INTEL_GEN(dev_priv) >= 8) { | 3201 | if (INTEL_GEN(dev_priv) >= 8) { |
| 3171 | hotplug_irqs = GEN8_PORT_DP_A_HOTPLUG; | 3202 | hotplug_irqs = GEN8_PORT_DP_A_HOTPLUG; |
| 3172 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_bdw); | 3203 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.hpd); |
| 3173 | 3204 | ||
| 3174 | bdw_update_port_irq(dev_priv, hotplug_irqs, enabled_irqs); | 3205 | bdw_update_port_irq(dev_priv, hotplug_irqs, enabled_irqs); |
| 3175 | } else if (INTEL_GEN(dev_priv) >= 7) { | 3206 | } else if (INTEL_GEN(dev_priv) >= 7) { |
| 3176 | hotplug_irqs = DE_DP_A_HOTPLUG_IVB; | 3207 | hotplug_irqs = DE_DP_A_HOTPLUG_IVB; |
| 3177 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_ivb); | 3208 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.hpd); |
| 3178 | 3209 | ||
| 3179 | ilk_update_display_irq(dev_priv, hotplug_irqs, enabled_irqs); | 3210 | ilk_update_display_irq(dev_priv, hotplug_irqs, enabled_irqs); |
| 3180 | } else { | 3211 | } else { |
| 3181 | hotplug_irqs = DE_DP_A_HOTPLUG; | 3212 | hotplug_irqs = DE_DP_A_HOTPLUG; |
| 3182 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_ilk); | 3213 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.hpd); |
| 3183 | 3214 | ||
| 3184 | ilk_update_display_irq(dev_priv, hotplug_irqs, enabled_irqs); | 3215 | ilk_update_display_irq(dev_priv, hotplug_irqs, enabled_irqs); |
| 3185 | } | 3216 | } |
| @@ -3230,7 +3261,7 @@ static void bxt_hpd_irq_setup(struct drm_i915_private *dev_priv) · 44 unmodified lines | |||
| 3230 | { | 3261 | { |
| 3231 | u32 hotplug_irqs, enabled_irqs; | 3262 | u32 hotplug_irqs, enabled_irqs; |
| 3232 | 3263 | ||
| 3233 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, hpd_bxt); | 3264 | enabled_irqs = intel_hpd_enabled_irqs(dev_priv, dev_priv->hotplug.hpd); |
| 3234 | hotplug_irqs = BXT_DE_PORT_HOTPLUG_MASK; | 3265 | hotplug_irqs = BXT_DE_PORT_HOTPLUG_MASK; |
| 3235 | 3266 | ||
| 3236 | bdw_update_port_irq(dev_priv, hotplug_irqs, enabled_irqs); | 3267 | bdw_update_port_irq(dev_priv, hotplug_irqs, enabled_irqs); |
| @@ -3361,7 +3392,7 @@ static void gen8_de_irq_postinstall(struct drm_i915_private *dev_priv) · 124 unmodified lines | |||
| 3361 | u32 de_pipe_masked = gen8_de_pipe_fault_mask(dev_priv) | | 3392 | u32 de_pipe_masked = gen8_de_pipe_fault_mask(dev_priv) | |
| 3362 | GEN8_PIPE_CDCLK_CRC_DONE; | 3393 | GEN8_PIPE_CDCLK_CRC_DONE; |
| 3363 | u32 de_pipe_enables; | 3394 | u32 de_pipe_enables; |
| 3364 | u32 de_port_masked = GEN8_AUX_CHANNEL_A; | 3395 | u32 de_port_masked = gen8_de_port_aux_mask(dev_priv); |
| 3365 | u32 de_port_enables; | 3396 | u32 de_port_enables; |
| 3366 | u32 de_misc_masked = GEN8_DE_EDP_PSR; | 3397 | u32 de_misc_masked = GEN8_DE_EDP_PSR; |
| 3367 | enum pipe pipe; | 3398 | enum pipe pipe; |
| @@ -3369,18 +3400,8 @@ static void gen8_de_irq_postinstall(struct drm_i915_private *dev_priv) · 1 unmodified line | |||
| 3369 | if (INTEL_GEN(dev_priv) <= 10) | 3400 | if (INTEL_GEN(dev_priv) <= 10) |
| 3370 | de_misc_masked |= GEN8_DE_MISC_GSE; | 3401 | de_misc_masked |= GEN8_DE_MISC_GSE; |
| 3371 | 3402 | ||
| 3372 | if (INTEL_GEN(dev_priv) >= 9) { | 3403 | if (IS_GEN9_LP(dev_priv)) |
| 3373 | de_port_masked |= GEN9_AUX_CHANNEL_B | GEN9_AUX_CHANNEL_C | | 3404 | de_port_masked |= BXT_DE_PORT_GMBUS; |
| 3374 | GEN9_AUX_CHANNEL_D; | ||
| 3375 | if (IS_GEN9_LP(dev_priv)) | ||
| 3376 | de_port_masked |= BXT_DE_PORT_GMBUS; | ||
| 3377 | } | ||
| 3378 | |||
| 3379 | if (INTEL_GEN(dev_priv) >= 11) | ||
| 3380 | de_port_masked |= ICL_AUX_CHANNEL_E; | ||
| 3381 | |||
| 3382 | if (IS_CNL_WITH_PORT_F(dev_priv) || INTEL_GEN(dev_priv) >= 11) | ||
| 3383 | de_port_masked |= CNL_AUX_CHANNEL_F; | ||
| 3384 | 3405 | ||
| 3385 | de_pipe_enables = de_pipe_masked | GEN8_PIPE_VBLANK | | 3406 | de_pipe_enables = de_pipe_masked | GEN8_PIPE_VBLANK | |
| 3386 | GEN8_PIPE_FIFO_UNDERRUN; | 3407 | GEN8_PIPE_FIFO_UNDERRUN; |
| @@ -3934,6 +3955,8 @@ void intel_irq_init(struct drm_i915_private *dev_priv) · 547 unmodified lines | |||
| 3934 | struct drm_device *dev = &dev_priv->drm; | 3955 | struct drm_device *dev = &dev_priv->drm; |
| 3935 | int i; | 3956 | int i; |
| 3936 | 3957 | ||
| 3958 | intel_hpd_init_pins(dev_priv); | ||
| 3959 | |||
| 3937 | intel_hpd_init_work(dev_priv); | 3960 | intel_hpd_init_work(dev_priv); |
| 3938 | 3961 | ||
| 3939 | INIT_WORK(&dev_priv->l3_parity.error_work, ivb_parity_work); | 3962 | INIT_WORK(&dev_priv->l3_parity.error_work, ivb_parity_work); |
| @@ -615,7 +615,7 @@ static const struct intel_device_info chv_info = { · 614 unmodified lines | |||
| 615 | .has_logical_ring_contexts = 1, | 615 | .has_logical_ring_contexts = 1, |
| 616 | .display.has_gmch = 1, | 616 | .display.has_gmch = 1, |
| 617 | .dma_mask_size = 39, | 617 | .dma_mask_size = 39, |
| 618 | .ppgtt_type = INTEL_PPGTT_ALIASING, | 618 | .ppgtt_type = INTEL_PPGTT_FULL, |
| 619 | .ppgtt_size = 32, | 619 | .ppgtt_size = 32, |
| 620 | .has_reset_engine = 1, | 620 | .has_reset_engine = 1, |
| 621 | .has_snoop = true, | 621 | .has_snoop = true, |
| @@ -1612,10 +1612,7 @@ static int alloc_noa_wait(struct i915_perf_stream *stream) · 1611 unmodified lines | |||
| 1612 | struct drm_i915_gem_object *bo; | 1612 | struct drm_i915_gem_object *bo; |
| 1613 | struct i915_vma *vma; | 1613 | struct i915_vma *vma; |
| 1614 | const u64 delay_ticks = 0xffffffffffffffff - | 1614 | const u64 delay_ticks = 0xffffffffffffffff - |
| 1615 | DIV64_U64_ROUND_UP( | 1615 | i915_cs_timestamp_ns_to_ticks(i915, atomic64_read(&stream->perf->noa_programming_delay)); |
| 1616 | atomic64_read(&stream->perf->noa_programming_delay) * | ||
| 1617 | RUNTIME_INFO(i915)->cs_timestamp_frequency_khz, | ||
| 1618 | 1000000ull); | ||
| 1619 | const u32 base = stream->engine->mmio_base; | 1616 | const u32 base = stream->engine->mmio_base; |
| 1620 | #define CS_GPR(x) GEN8_RING_CS_GPR(base, x) | 1617 | #define CS_GPR(x) GEN8_RING_CS_GPR(base, x) |
| 1621 | u32 *batch, *ts0, *cs, *jump; | 1618 | u32 *batch, *ts0, *cs, *jump; |
| @@ -3485,8 +3482,7 @@ err: · 1863 unmodified lines | |||
| 3485 | 3482 | ||
| 3486 | static u64 oa_exponent_to_ns(struct i915_perf *perf, int exponent) | 3483 | static u64 oa_exponent_to_ns(struct i915_perf *perf, int exponent) |
| 3487 | { | 3484 | { |
| 3488 | return div64_u64(1000000000ULL * (2ULL << exponent), | 3485 | return i915_cs_timestamp_ticks_to_ns(perf->i915, 2ULL << exponent); |
| 3489 | 1000ULL * RUNTIME_INFO(perf->i915)->cs_timestamp_frequency_khz); | ||
| 3490 | } | 3486 | } |
| 3491 | 3487 | ||
| 3492 | /** | 3488 | /** |
| @@ -4344,8 +4340,8 @@ void i915_perf_init(struct drm_i915_private *i915) · 851 unmodified lines | |||
| 4344 | if (perf->ops.enable_metric_set) { | 4340 | if (perf->ops.enable_metric_set) { |
| 4345 | mutex_init(&perf->lock); | 4341 | mutex_init(&perf->lock); |
| 4346 | 4342 | ||
| 4347 | oa_sample_rate_hard_limit = 1000 * | 4343 | oa_sample_rate_hard_limit = |
| 4348 | (RUNTIME_INFO(i915)->cs_timestamp_frequency_khz / 2); | 4344 | RUNTIME_INFO(i915)->cs_timestamp_frequency_hz / 2; |
| 4349 | 4345 | ||
| 4350 | mutex_init(&perf->metrics_lock); | 4346 | mutex_init(&perf->metrics_lock); |
| 4351 | idr_init(&perf->metrics_idr); | 4347 | idr_init(&perf->metrics_idr); |
| @@ -442,6 +442,7 @@ static u64 count_interrupts(struct drm_i915_private *i915) · 441 unmodified lines | |||
| 442 | static void i915_pmu_event_destroy(struct perf_event *event) | 442 | static void i915_pmu_event_destroy(struct perf_event *event) |
| 443 | { | 443 | { |
| 444 | WARN_ON(event->parent); | 444 | WARN_ON(event->parent); |
| 445 | module_put(THIS_MODULE); | ||
| 445 | } | 446 | } |
| 446 | 447 | ||
| 447 | static int | 448 | static int |
| @@ -533,8 +534,10 @@ static int i915_pmu_event_init(struct perf_event *event) · 85 unmodified lines | |||
| 533 | if (ret) | 534 | if (ret) |
| 534 | return ret; | 535 | return ret; |
| 535 | 536 | ||
| 536 | if (!event->parent) | 537 | if (!event->parent) { |
| 538 | __module_get(THIS_MODULE); | ||
| 537 | event->destroy = i915_pmu_event_destroy; | 539 | event->destroy = i915_pmu_event_destroy; |
| 540 | } | ||
| 538 | 541 | ||
| 539 | return 0; | 542 | return 0; |
| 540 | } | 543 | } |
| @@ -24,15 +24,12 @@ enum { · 23 unmodified lines | |||
| 24 | I915_PRIORITY_DISPLAY, | 24 | I915_PRIORITY_DISPLAY, |
| 25 | }; | 25 | }; |
| 26 | 26 | ||
| 27 | #define I915_USER_PRIORITY_SHIFT 2 | 27 | #define I915_USER_PRIORITY_SHIFT 0 |
| 28 | #define I915_USER_PRIORITY(x) ((x) << I915_USER_PRIORITY_SHIFT) | 28 | #define I915_USER_PRIORITY(x) ((x) << I915_USER_PRIORITY_SHIFT) |
| 29 | 29 | ||
| 30 | #define I915_PRIORITY_COUNT BIT(I915_USER_PRIORITY_SHIFT) | 30 | #define I915_PRIORITY_COUNT BIT(I915_USER_PRIORITY_SHIFT) |
| 31 | #define I915_PRIORITY_MASK (I915_PRIORITY_COUNT - 1) | 31 | #define I915_PRIORITY_MASK (I915_PRIORITY_COUNT - 1) |
| 32 | 32 | ||
| 33 | #define I915_PRIORITY_WAIT ((u8)BIT(0)) | ||
| 34 | #define I915_PRIORITY_NOSEMAPHORE ((u8)BIT(1)) | ||
| 35 | |||
| 36 | /* Smallest priority value that cannot be bumped. */ | 33 | /* Smallest priority value that cannot be bumped. */ |
| 37 | #define I915_PRIORITY_INVALID (INT_MIN | (u8)I915_PRIORITY_MASK) | 34 | #define I915_PRIORITY_INVALID (INT_MIN | (u8)I915_PRIORITY_MASK) |
| 38 | 35 | ||
| @@ -47,8 +44,6 @@ enum { · 8 unmodified lines | |||
| 47 | #define I915_PRIORITY_UNPREEMPTABLE INT_MAX | 44 | #define I915_PRIORITY_UNPREEMPTABLE INT_MAX |
| 48 | #define I915_PRIORITY_BARRIER INT_MAX | 45 | #define I915_PRIORITY_BARRIER INT_MAX |
| 49 | 46 | ||
| 50 | #define __NO_PREEMPTION (I915_PRIORITY_WAIT) | ||
| 51 | |||
| 52 | struct i915_priolist { | 47 | struct i915_priolist { |
| 53 | struct list_head requests[I915_PRIORITY_COUNT]; | 48 | struct list_head requests[I915_PRIORITY_COUNT]; |
| 54 | struct rb_node node; | 49 | struct rb_node node; |
| @@ -2557,6 +2557,14 @@ static inline bool i915_mmio_reg_valid(i915_reg_t reg) · 2556 unmodified lines | |||
| 2557 | #define GEN10_PAT_INDEX(index) _MMIO(0x40e0 + (index) * 4) | 2557 | #define GEN10_PAT_INDEX(index) _MMIO(0x40e0 + (index) * 4) |
| 2558 | #define GEN12_PAT_INDEX(index) _MMIO(0x4800 + (index) * 4) | 2558 | #define GEN12_PAT_INDEX(index) _MMIO(0x4800 + (index) * 4) |
| 2559 | #define BSD_HWS_PGA_GEN7 _MMIO(0x04180) | 2559 | #define BSD_HWS_PGA_GEN7 _MMIO(0x04180) |
| 2560 | #define GEN12_GFX_CCS_AUX_NV _MMIO(0x4208) | ||
| 2561 | #define GEN12_VD0_AUX_NV _MMIO(0x4218) | ||
| 2562 | #define GEN12_VD1_AUX_NV _MMIO(0x4228) | ||
| 2563 | #define GEN12_VD2_AUX_NV _MMIO(0x4298) | ||
| 2564 | #define GEN12_VD3_AUX_NV _MMIO(0x42A8) | ||
| 2565 | #define GEN12_VE0_AUX_NV _MMIO(0x4238) | ||
| 2566 | #define GEN12_VE1_AUX_NV _MMIO(0x42B8) | ||
| 2567 | #define AUX_INV REG_BIT(0) | ||
| 2560 | #define BLT_HWS_PGA_GEN7 _MMIO(0x04280) | 2568 | #define BLT_HWS_PGA_GEN7 _MMIO(0x04280) |
| 2561 | #define VEBOX_HWS_PGA_GEN7 _MMIO(0x04380) | 2569 | #define VEBOX_HWS_PGA_GEN7 _MMIO(0x04380) |
| 2562 | #define RING_ACTHD(base) _MMIO((base) + 0x74) | 2570 | #define RING_ACTHD(base) _MMIO((base) + 0x74) |
| @@ -8573,6 +8581,7 @@ enum { · 6010 unmodified lines | |||
| 8573 | #define FDI_BC_BIFURCATION_SELECT (1 << 12) | 8581 | #define FDI_BC_BIFURCATION_SELECT (1 << 12) |
| 8574 | #define CHASSIS_CLK_REQ_DURATION_MASK (0xf << 8) | 8582 | #define CHASSIS_CLK_REQ_DURATION_MASK (0xf << 8) |
| 8575 | #define CHASSIS_CLK_REQ_DURATION(x) ((x) << 8) | 8583 | #define CHASSIS_CLK_REQ_DURATION(x) ((x) << 8) |
| 8584 | #define SBCLK_RUN_REFCLK_DIS (1 << 7) | ||
| 8576 | #define SPT_PWM_GRANULARITY (1 << 0) | 8585 | #define SPT_PWM_GRANULARITY (1 << 0) |
| 8577 | #define SOUTH_CHICKEN2 _MMIO(0xc2004) | 8586 | #define SOUTH_CHICKEN2 _MMIO(0xc2004) |
| 8578 | #define FDI_MPHY_IOSFSB_RESET_STATUS (1 << 13) | 8587 | #define FDI_MPHY_IOSFSB_RESET_STATUS (1 << 13) |
| @@ -9064,6 +9073,7 @@ enum { · 485 unmodified lines | |||
| 9064 | #define GEN7_PCODE_ILLEGAL_DATA 0x3 | 9073 | #define GEN7_PCODE_ILLEGAL_DATA 0x3 |
| 9065 | #define GEN11_PCODE_ILLEGAL_SUBCOMMAND 0x4 | 9074 | #define GEN11_PCODE_ILLEGAL_SUBCOMMAND 0x4 |
| 9066 | #define GEN11_PCODE_LOCKED 0x6 | 9075 | #define GEN11_PCODE_LOCKED 0x6 |
| 9076 | #define GEN11_PCODE_REJECTED 0x11 | ||
| 9067 | #define GEN7_PCODE_MIN_FREQ_TABLE_GT_RATIO_OUT_OF_RANGE 0x10 | 9077 | #define GEN7_PCODE_MIN_FREQ_TABLE_GT_RATIO_OUT_OF_RANGE 0x10 |
| 9068 | #define GEN6_PCODE_WRITE_RC6VIDS 0x4 | 9078 | #define GEN6_PCODE_WRITE_RC6VIDS 0x4 |
| 9069 | #define GEN6_PCODE_READ_RC6VIDS 0x5 | 9079 | #define GEN6_PCODE_READ_RC6VIDS 0x5 |
| @@ -9085,6 +9095,9 @@ enum { · 15 unmodified lines | |||
| 9085 | #define ICL_PCODE_MEM_SUBSYSYSTEM_INFO 0xd | 9095 | #define ICL_PCODE_MEM_SUBSYSYSTEM_INFO 0xd |
| 9086 | #define ICL_PCODE_MEM_SS_READ_GLOBAL_INFO (0x0 << 8) | 9096 | #define ICL_PCODE_MEM_SS_READ_GLOBAL_INFO (0x0 << 8) |
| 9087 | #define ICL_PCODE_MEM_SS_READ_QGV_POINT_INFO(point) (((point) << 16) | (0x1 << 8)) | 9097 | #define ICL_PCODE_MEM_SS_READ_QGV_POINT_INFO(point) (((point) << 16) | (0x1 << 8)) |
| 9098 | #define ICL_PCODE_SAGV_DE_MEM_SS_CONFIG 0xe | ||
| 9099 | #define ICL_PCODE_POINTS_RESTRICTED 0x0 | ||
| 9100 | #define ICL_PCODE_POINTS_RESTRICTED_MASK 0x1 | ||
| 9088 | #define GEN6_PCODE_READ_D_COMP 0x10 | 9101 | #define GEN6_PCODE_READ_D_COMP 0x10 |
| 9089 | #define GEN6_PCODE_WRITE_D_COMP 0x11 | 9102 | #define GEN6_PCODE_WRITE_D_COMP 0x11 |
| 9090 | #define ICL_PCODE_EXIT_TCCOLD 0x12 | 9103 | #define ICL_PCODE_EXIT_TCCOLD 0x12 |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | |||
| 23 | */ | 23 | */ |
| 24 | 24 | ||
| 25 | #include <linux/dma-fence-array.h> | 25 | #include <linux/dma-fence-array.h> |
| 26 | #include <linux/dma-fence-chain.h> | ||
| 26 | #include <linux/irq_work.h> | 27 | #include <linux/irq_work.h> |
| 27 | #include <linux/prefetch.h> | 28 | #include <linux/prefetch.h> |
| 28 | #include <linux/sched.h> | 29 | #include <linux/sched.h> |
| @@ -367,8 +368,6 @@ __await_execution(struct i915_request *rq, · 338 unmodified lines | |||
| 367 | } | 368 | } |
| 368 | spin_unlock_irq(&signal->lock); | 369 | spin_unlock_irq(&signal->lock); |
| 369 | 370 | ||
| 370 | /* Copy across semaphore status as we need the same behaviour */ | ||
| 371 | rq->sched.flags |= signal->sched.flags; | ||
| 372 | return 0; | 371 | return 0; |
| 373 | } | 372 | } |
| 374 | 373 | ||
| @@ -536,10 +535,8 @@ void __i915_request_unsubmit(struct i915_request *request) · 161 unmodified lines | |||
| 536 | spin_unlock(&request->lock); | 535 | spin_unlock(&request->lock); |
| 537 | 536 | ||
| 538 | /* We've already spun, don't charge on resubmitting. */ | 537 | /* We've already spun, don't charge on resubmitting. */ |
| 539 | if (request->sched.semaphores && i915_request_started(request)) { | 538 | if (request->sched.semaphores && i915_request_started(request)) |
| 540 | request->sched.attr.priority |= I915_PRIORITY_NOSEMAPHORE; | ||
| 541 | request->sched.semaphores = 0; | 539 | request->sched.semaphores = 0; |
| 542 | } | ||
| 543 | 540 | ||
| 544 | /* | 541 | /* |
| 545 | * We don't need to wake_up any waiters on request->execute, they | 542 | * We don't need to wake_up any waiters on request->execute, they |
| @@ -597,15 +594,6 @@ submit_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state) · 51 unmodified lines | |||
| 597 | return NOTIFY_DONE; | 594 | return NOTIFY_DONE; |
| 598 | } | 595 | } |
| 599 | 596 | ||
| 600 | static void irq_semaphore_cb(struct irq_work *wrk) | ||
| 601 | { | ||
| 602 | struct i915_request *rq = | ||
| 603 | container_of(wrk, typeof(*rq), semaphore_work); | ||
| 604 | |||
| 605 | i915_schedule_bump_priority(rq, I915_PRIORITY_NOSEMAPHORE); | ||
| 606 | i915_request_put(rq); | ||
| 607 | } | ||
| 608 | |||
| 609 | static int __i915_sw_fence_call | 597 | static int __i915_sw_fence_call |
| 610 | semaphore_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state) | 598 | semaphore_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state) |
| 611 | { | 599 | { |
| @@ -613,11 +601,6 @@ semaphore_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state) · 1 unmodified line | |||
| 613 | 601 | ||
| 614 | switch (state) { | 602 | switch (state) { |
| 615 | case FENCE_COMPLETE: | 603 | case FENCE_COMPLETE: |
| 616 | if (!(READ_ONCE(rq->sched.attr.priority) & I915_PRIORITY_NOSEMAPHORE)) { | ||
| 617 | i915_request_get(rq); | ||
| 618 | init_irq_work(&rq->semaphore_work, irq_semaphore_cb); | ||
| 619 | irq_work_queue(&rq->semaphore_work); | ||
| 620 | } | ||
| 621 | break; | 604 | break; |
| 622 | 605 | ||
| 623 | case FENCE_FREE: | 606 | case FENCE_FREE: |
| @@ -950,6 +933,7 @@ __emit_semaphore_wait(struct i915_request *to, · 326 unmodified lines | |||
| 950 | u32 *cs; | 933 | u32 *cs; |
| 951 | 934 | ||
| 952 | GEM_BUG_ON(INTEL_GEN(to->i915) < 8); | 935 | GEM_BUG_ON(INTEL_GEN(to->i915) < 8); |
| 936 | GEM_BUG_ON(i915_request_has_initial_breadcrumb(to)); | ||
| 953 | 937 | ||
| 954 | /* We need to pin the signaler's HWSP until we are finished reading. */ | 938 | /* We need to pin the signaler's HWSP until we are finished reading. */ |
| 955 | err = intel_timeline_read_hwsp(from, to, &hwsp_offset); | 939 | err = intel_timeline_read_hwsp(from, to, &hwsp_offset); |
| @@ -995,13 +979,26 @@ emit_semaphore_wait(struct i915_request *to, · 39 unmodified lines | |||
| 995 | gfp_t gfp) | 979 | gfp_t gfp) |
| 996 | { | 980 | { |
| 997 | const intel_engine_mask_t mask = READ_ONCE(from->engine)->mask; | 981 | const intel_engine_mask_t mask = READ_ONCE(from->engine)->mask; |
| 982 | struct i915_sw_fence *wait = &to->submit; | ||
| 998 | 983 | ||
| 999 | if (!intel_context_use_semaphores(to->context)) | 984 | if (!intel_context_use_semaphores(to->context)) |
| 1000 | goto await_fence; | 985 | goto await_fence; |
| 1001 | 986 | ||
| 987 | if (i915_request_has_initial_breadcrumb(to)) | ||
| 988 | goto await_fence; | ||
| 989 | |||
| 1002 | if (!rcu_access_pointer(from->hwsp_cacheline)) | 990 | if (!rcu_access_pointer(from->hwsp_cacheline)) |
| 1003 | goto await_fence; | 991 | goto await_fence; |
| 1004 | 992 | ||
| 993 | /* | ||
| 994 | * If this or its dependents are waiting on an external fence | ||
| 995 | * that may fail catastrophically, then we want to avoid using | ||
| 996 | * sempahores as they bypass the fence signaling metadata, and we | ||
| 997 | * lose the fence->error propagation. | ||
| 998 | */ | ||
| 999 | if (from->sched.flags & I915_SCHED_HAS_EXTERNAL_CHAIN) | ||
| 1000 | goto await_fence; | ||
| 1001 | |||
| 1005 | /* Just emit the first semaphore we see as request space is limited. */ | 1002 | /* Just emit the first semaphore we see as request space is limited. */ |
| 1006 | if (already_busywaiting(to) & mask) | 1003 | if (already_busywaiting(to) & mask) |
| 1007 | goto await_fence; | 1004 | goto await_fence; |
| @@ -1017,11 +1014,10 @@ emit_semaphore_wait(struct i915_request *to, · 9 unmodified lines | |||
| 1017 | goto await_fence; | 1014 | goto await_fence; |
| 1018 | 1015 | ||
| 1019 | to->sched.semaphores |= mask; | 1016 | to->sched.semaphores |= mask; |
| 1020 | to->sched.flags |= I915_SCHED_HAS_SEMAPHORE_CHAIN; | 1017 | wait = &to->semaphore; |
| 1021 | return 0; | ||
| 1022 | 1018 | ||
| 1023 | await_fence: | 1019 | await_fence: |
| 1024 | return i915_sw_fence_await_dma_fence(&to->submit, | 1020 | return i915_sw_fence_await_dma_fence(wait, |
| 1025 | &from->fence, 0, | 1021 | &from->fence, 0, |
| 1026 | I915_FENCE_GFP); | 1022 | I915_FENCE_GFP); |
| 1027 | } | 1023 | } |
| @@ -1034,11 +1030,15 @@ i915_request_await_request(struct i915_request *to, struct i915_request *from) · 6 unmodified lines | |||
| 1034 | GEM_BUG_ON(to == from); | 1030 | GEM_BUG_ON(to == from); |
| 1035 | GEM_BUG_ON(to->timeline == from->timeline); | 1031 | GEM_BUG_ON(to->timeline == from->timeline); |
| 1036 | 1032 | ||
| 1037 | if (i915_request_completed(from)) | 1033 | if (i915_request_completed(from)) { |
| 1034 | i915_sw_fence_set_error_once(&to->submit, from->fence.error); | ||
| 1038 | return 0; | 1035 | return 0; |
| 1036 | } | ||
| 1039 | 1037 | ||
| 1040 | if (to->engine->schedule) { | 1038 | if (to->engine->schedule) { |
| 1041 | ret = i915_sched_node_add_dependency(&to->sched, &from->sched); | 1039 | ret = i915_sched_node_add_dependency(&to->sched, |
| 1040 | &from->sched, | ||
| 1041 | I915_DEPENDENCY_EXTERNAL); | ||
| 1042 | if (ret < 0) | 1042 | if (ret < 0) |
| 1043 | return ret; | 1043 | return ret; |
| 1044 | } | 1044 | } |
| @@ -1052,15 +1052,56 @@ i915_request_await_request(struct i915_request *to, struct i915_request *from) · 7 unmodified lines | |||
| 1052 | if (ret < 0) | 1052 | if (ret < 0) |
| 1053 | return ret; | 1053 | return ret; |
| 1054 | 1054 | ||
| 1055 | if (to->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN) { | 1055 | return 0; |
| 1056 | ret = i915_sw_fence_await_dma_fence(&to->semaphore, | 1056 | } |
| 1057 | &from->fence, 0, | 1057 | |
| 1058 | I915_FENCE_GFP); | 1058 | static void mark_external(struct i915_request *rq) |
| 1059 | if (ret < 0) | 1059 | { |
| 1060 | return ret; | 1060 | /* |
| 1061 | * The downside of using semaphores is that we lose metadata passing | ||
| 1062 | * along the signaling chain. This is particularly nasty when we | ||
| 1063 | * need to pass along a fatal error such as EFAULT or EDEADLK. For | ||
| 1064 | * fatal errors we want to scrub the request before it is executed, | ||
| 1065 | * which means that we cannot preload the request onto HW and have | ||
| 1066 | * it wait upon a semaphore. | ||
| 1067 | */ | ||
| 1068 | rq->sched.flags |= I915_SCHED_HAS_EXTERNAL_CHAIN; | ||
| 1069 | } | ||
| 1070 | |||
| 1071 | static int | ||
| 1072 | __i915_request_await_external(struct i915_request *rq, struct dma_fence *fence) | ||
| 1073 | { | ||
| 1074 | mark_external(rq); | ||
| 1075 | return i915_sw_fence_await_dma_fence(&rq->submit, fence, | ||
| 1076 | i915_fence_context_timeout(rq->i915, | ||
| 1077 | fence->context), | ||
| 1078 | I915_FENCE_GFP); | ||
| 1079 | } | ||
| 1080 | |||
| 1081 | static int | ||
| 1082 | i915_request_await_external(struct i915_request *rq, struct dma_fence *fence) | ||
| 1083 | { | ||
| 1084 | struct dma_fence *iter; | ||
| 1085 | int err = 0; | ||
| 1086 | |||
| 1087 | if (!to_dma_fence_chain(fence)) | ||
| 1088 | return __i915_request_await_external(rq, fence); | ||
| 1089 | |||
| 1090 | dma_fence_chain_for_each(iter, fence) { | ||
| 1091 | struct dma_fence_chain *chain = to_dma_fence_chain(iter); | ||
| 1092 | |||
| 1093 | if (!dma_fence_is_i915(chain->fence)) { | ||
| 1094 | err = __i915_request_await_external(rq, iter); | ||
| 1095 | break; | ||
| 1096 | } | ||
| 1097 | |||
| 1098 | err = i915_request_await_dma_fence(rq, chain->fence); | ||
| 1099 | if (err < 0) | ||
| 1100 | break; | ||
| 1061 | } | 1101 | } |
| 1062 | 1102 | ||
| 1063 | return 0; | 1103 | dma_fence_put(iter); |
| 1104 | return err; | ||
| 1064 | } | 1105 | } |
| 1065 | 1106 | ||
| 1066 | int | 1107 | int |
| @@ -1110,9 +1151,7 @@ i915_request_await_dma_fence(struct i915_request *rq, struct dma_fence *fence) · 43 unmodified lines | |||
| 1110 | if (dma_fence_is_i915(fence)) | 1151 | if (dma_fence_is_i915(fence)) |
| 1111 | ret = i915_request_await_request(rq, to_request(fence)); | 1152 | ret = i915_request_await_request(rq, to_request(fence)); |
| 1112 | else | 1153 | else |
| 1113 | ret = i915_sw_fence_await_dma_fence(&rq->submit, fence, | 1154 | ret = i915_request_await_external(rq, fence); |
| 1114 | fence->context ? I915_FENCE_TIMEOUT : 0, | ||
| 1115 | I915_FENCE_GFP); | ||
| 1116 | if (ret < 0) | 1155 | if (ret < 0) |
| 1117 | return ret; | 1156 | return ret; |
| 1118 | 1157 | ||
| @@ -1192,7 +1231,8 @@ __i915_request_await_execution(struct i915_request *to, · 73 unmodified lines | |||
| 1192 | * immediate execution, and so we must wait until it reaches the | 1231 | * immediate execution, and so we must wait until it reaches the |
| 1193 | * active slot. | 1232 | * active slot. |
| 1194 | */ | 1233 | */ |
| 1195 | if (intel_engine_has_semaphores(to->engine)) { | 1234 | if (intel_engine_has_semaphores(to->engine) && |
| 1235 | !i915_request_has_initial_breadcrumb(to)) { | ||
| 1196 | err = __emit_semaphore_wait(to, from, from->fence.seqno - 1); | 1236 | err = __emit_semaphore_wait(to, from, from->fence.seqno - 1); |
| 1197 | if (err < 0) | 1237 | if (err < 0) |
| 1198 | return err; | 1238 | return err; |
| @@ -1200,7 +1240,9 @@ __i915_request_await_execution(struct i915_request *to, · 1 unmodified line | |||
| 1200 | 1240 | ||
| 1201 | /* Couple the dependency tree for PI on this exposed to->fence */ | 1241 | /* Couple the dependency tree for PI on this exposed to->fence */ |
| 1202 | if (to->engine->schedule) { | 1242 | if (to->engine->schedule) { |
| 1203 | err = i915_sched_node_add_dependency(&to->sched, &from->sched); | 1243 | err = i915_sched_node_add_dependency(&to->sched, |
| 1244 | &from->sched, | ||
| 1245 | I915_DEPENDENCY_WEAK); | ||
| 1204 | if (err < 0) | 1246 | if (err < 0) |
| 1205 | return err; | 1247 | return err; |
| 1206 | } | 1248 | } |
| @@ -1236,6 +1278,9 @@ i915_request_await_execution(struct i915_request *rq, · 29 unmodified lines | |||
| 1236 | continue; | 1278 | continue; |
| 1237 | } | 1279 | } |
| 1238 | 1280 | ||
| 1281 | if (fence->context == rq->fence.context) | ||
| 1282 | continue; | ||
| 1283 | |||
| 1239 | /* | 1284 | /* |
| 1240 | * We don't squash repeated fence dependencies here as we | 1285 | * We don't squash repeated fence dependencies here as we |
| 1241 | * want to run our callback in all cases. | 1286 | * want to run our callback in all cases. |
| @@ -1246,9 +1291,7 @@ i915_request_await_execution(struct i915_request *rq, · 4 unmodified lines | |||
| 1246 | to_request(fence), | 1291 | to_request(fence), |
| 1247 | hook); | 1292 | hook); |
| 1248 | else | 1293 | else |
| 1249 | ret = i915_sw_fence_await_dma_fence(&rq->submit, fence, | 1294 | ret = i915_request_await_external(rq, fence); |
| 1250 | I915_FENCE_TIMEOUT, | ||
| 1251 | GFP_KERNEL); | ||
| 1252 | if (ret < 0) | 1295 | if (ret < 0) |
| 1253 | return ret; | 1296 | return ret; |
| 1254 | } while (--nchild); | 1297 | } while (--nchild); |
| @@ -1456,11 +1499,6 @@ void i915_request_add(struct i915_request *rq) · 201 unmodified lines | |||
| 1456 | attr = ctx->sched; | 1499 | attr = ctx->sched; |
| 1457 | rcu_read_unlock(); | 1500 | rcu_read_unlock(); |
| 1458 | 1501 | ||
| 1459 | if (!(rq->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN)) | ||
| 1460 | attr.priority |= I915_PRIORITY_NOSEMAPHORE; | ||
| 1461 | if (list_empty(&rq->sched.signalers_list)) | ||
| 1462 | attr.priority |= I915_PRIORITY_WAIT; | ||
| 1463 | |||
| 1464 | __i915_request_queue(rq, &attr); | 1502 | __i915_request_queue(rq, &attr); |
| 1465 | 1503 | ||
| 1466 | mutex_unlock(&tl->mutex); | 1504 | mutex_unlock(&tl->mutex); |
| @@ -1645,7 +1683,6 @@ long i915_request_wait(struct i915_request *rq, · 178 unmodified lines | |||
| 1645 | if (flags & I915_WAIT_PRIORITY) { | 1683 | if (flags & I915_WAIT_PRIORITY) { |
| 1646 | if (!i915_request_started(rq) && INTEL_GEN(rq->i915) >= 6) | 1684 | if (!i915_request_started(rq) && INTEL_GEN(rq->i915) >= 6) |
| 1647 | intel_rps_boost(rq); | 1685 | intel_rps_boost(rq); |
| 1648 | i915_schedule_bump_priority(rq, I915_PRIORITY_WAIT); | ||
| 1649 | } | 1686 | } |
| 1650 | 1687 | ||
| 1651 | wait.tsk = current; | 1688 | wait.tsk = current; |
| @@ -84,19 +84,26 @@ enum { · 83 unmodified lines | |||
| 84 | I915_FENCE_FLAG_PQUEUE, | 84 | I915_FENCE_FLAG_PQUEUE, |
| 85 | 85 | ||
| 86 | /* | 86 | /* |
| 87 | * I915_FENCE_FLAG_SIGNAL - this request is currently on signal_list | 87 | * I915_FENCE_FLAG_HOLD - this request is currently on hold |
| 88 | * | 88 | * |
| 89 | * Internal bookkeeping used by the breadcrumb code to track when | 89 | * This request has been suspended, pending an ongoing investigation. |
| 90 | * a request is on the various signal_list. | ||
| 91 | */ | 90 | */ |
| 92 | I915_FENCE_FLAG_SIGNAL, | 91 | I915_FENCE_FLAG_HOLD, |
| 93 | 92 | ||
| 94 | /* | 93 | /* |
| 95 | * I915_FENCE_FLAG_HOLD - this request is currently on hold | 94 | * I915_FENCE_FLAG_INITIAL_BREADCRUMB - this request has the initial |
| 95 | * breadcrumb that marks the end of semaphore waits and start of the | ||
| 96 | * user payload. | ||
| 97 | */ | ||
| 98 | I915_FENCE_FLAG_INITIAL_BREADCRUMB, | ||
| 99 | |||
| 100 | /* | ||
| 101 | * I915_FENCE_FLAG_SIGNAL - this request is currently on signal_list | ||
| 96 | * | 102 | * |
| 97 | * This request has been suspended, pending an ongoing investigation. | 103 | * Internal bookkeeping used by the breadcrumb code to track when |
| 104 | * a request is on the various signal_list. | ||
| 98 | */ | 105 | */ |
| 99 | I915_FENCE_FLAG_HOLD, | 106 | I915_FENCE_FLAG_SIGNAL, |
| 100 | 107 | ||
| 101 | /* | 108 | /* |
| 102 | * I915_FENCE_FLAG_NOPREEMPT - this request should not be preempted | 109 | * I915_FENCE_FLAG_NOPREEMPT - this request should not be preempted |
| @@ -209,7 +216,6 @@ struct i915_request { · 106 unmodified lines | |||
| 209 | }; | 216 | }; |
| 210 | struct list_head execute_cb; | 217 | struct list_head execute_cb; |
| 211 | struct i915_sw_fence semaphore; | 218 | struct i915_sw_fence semaphore; |
| 212 | struct irq_work semaphore_work; | ||
| 213 | 219 | ||
| 214 | /* | 220 | /* |
| 215 | * A list of everyone we wait upon, and everyone who waits upon us. | 221 | * A list of everyone we wait upon, and everyone who waits upon us. |
| @@ -390,6 +396,12 @@ static inline bool i915_request_in_priority_queue(const struct i915_request *rq) · 174 unmodified lines | |||
| 390 | return test_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags); | 396 | return test_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags); |
| 391 | } | 397 | } |
| 392 | 398 | ||
| 399 | static inline bool | ||
| 400 | i915_request_has_initial_breadcrumb(const struct i915_request *rq) | ||
| 401 | { | ||
| 402 | return test_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags); | ||
| 403 | } | ||
| 404 | |||
| 393 | /** | 405 | /** |
| 394 | * Returns true if seq1 is later than seq2. | 406 | * Returns true if seq1 is later than seq2. |
| 395 | */ | 407 | */ |
| @@ -51,11 +51,11 @@ static void assert_priolists(struct intel_engine_execlists * const execlists) · 50 unmodified lines | |||
| 51 | GEM_BUG_ON(rb_first_cached(&execlists->queue) != | 51 | GEM_BUG_ON(rb_first_cached(&execlists->queue) != |
| 52 | rb_first(&execlists->queue.rb_root)); | 52 | rb_first(&execlists->queue.rb_root)); |
| 53 | 53 | ||
| 54 | last_prio = (INT_MAX >> I915_USER_PRIORITY_SHIFT) + 1; | 54 | last_prio = INT_MAX; |
| 55 | for (rb = rb_first_cached(&execlists->queue); rb; rb = rb_next(rb)) { | 55 | for (rb = rb_first_cached(&execlists->queue); rb; rb = rb_next(rb)) { |
| 56 | const struct i915_priolist *p = to_priolist(rb); | 56 | const struct i915_priolist *p = to_priolist(rb); |
| 57 | 57 | ||
| 58 | GEM_BUG_ON(p->priority >= last_prio); | 58 | GEM_BUG_ON(p->priority > last_prio); |
| 59 | last_prio = p->priority; | 59 | last_prio = p->priority; |
| 60 | 60 | ||
| 61 | GEM_BUG_ON(!p->used); | 61 | GEM_BUG_ON(!p->used); |
| @@ -174,7 +174,7 @@ sched_lock_engine(const struct i915_sched_node *node, · 112 unmodified lines | |||
| 174 | 174 | ||
| 175 | static inline int rq_prio(const struct i915_request *rq) | 175 | static inline int rq_prio(const struct i915_request *rq) |
| 176 | { | 176 | { |
| 177 | return rq->sched.attr.priority | __NO_PREEMPTION; | 177 | return rq->sched.attr.priority; |
| 178 | } | 178 | } |
| 179 | 179 | ||
| 180 | static inline bool need_preempt(int prio, int active) | 180 | static inline bool need_preempt(int prio, int active) |
| @@ -434,25 +434,12 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, · 253 unmodified lines | |||
| 434 | dep->waiter = node; | 434 | dep->waiter = node; |
| 435 | dep->flags = flags; | 435 | dep->flags = flags; |
| 436 | 436 | ||
| 437 | /* Keep track of whether anyone on this chain has a semaphore */ | ||
| 438 | if (signal->flags & I915_SCHED_HAS_SEMAPHORE_CHAIN && | ||
| 439 | !node_started(signal)) | ||
| 440 | node->flags |= I915_SCHED_HAS_SEMAPHORE_CHAIN; | ||
| 441 | |||
| 442 | /* All set, now publish. Beware the lockless walkers. */ | 437 | /* All set, now publish. Beware the lockless walkers. */ |
| 443 | list_add_rcu(&dep->signal_link, &node->signalers_list); | 438 | list_add_rcu(&dep->signal_link, &node->signalers_list); |
| 444 | list_add_rcu(&dep->wait_link, &signal->waiters_list); | 439 | list_add_rcu(&dep->wait_link, &signal->waiters_list); |
| 445 | 440 | ||
| 446 | /* | 441 | /* Propagate the chains */ |
| 447 | * As we do not allow WAIT to preempt inflight requests, | 442 | node->flags |= signal->flags; |
| 448 | * once we have executed a request, along with triggering | ||
| 449 | * any execution callbacks, we must preserve its ordering | ||
| 450 | * within the non-preemptible FIFO. | ||
| 451 | */ | ||
| 452 | BUILD_BUG_ON(__NO_PREEMPTION & ~I915_PRIORITY_MASK); | ||
| 453 | if (flags & I915_DEPENDENCY_EXTERNAL) | ||
| 454 | __bump_priority(signal, __NO_PREEMPTION); | ||
| 455 | |||
| 456 | ret = true; | 443 | ret = true; |
| 457 | } | 444 | } |
| 458 | 445 | ||
| @@ -462,7 +449,8 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, · 3 unmodified lines | |||
| 462 | } | 449 | } |
| 463 | 450 | ||
| 464 | int i915_sched_node_add_dependency(struct i915_sched_node *node, | 451 | int i915_sched_node_add_dependency(struct i915_sched_node *node, |
| 465 | struct i915_sched_node *signal) | 452 | struct i915_sched_node *signal, |
| 453 | unsigned long flags) | ||
| 466 | { | 454 | { |
| 467 | struct i915_dependency *dep; | 455 | struct i915_dependency *dep; |
| 468 | 456 | ||
| @@ -473,8 +461,7 @@ int i915_sched_node_add_dependency(struct i915_sched_node *node, · 4 unmodified lines | |||
| 473 | local_bh_disable(); | 461 | local_bh_disable(); |
| 474 | 462 | ||
| 475 | if (!__i915_sched_node_add_dependency(node, signal, dep, | 463 | if (!__i915_sched_node_add_dependency(node, signal, dep, |
| 476 | I915_DEPENDENCY_EXTERNAL | | 464 | flags | I915_DEPENDENCY_ALLOC)) |
| 477 | I915_DEPENDENCY_ALLOC)) | ||
| 478 | i915_dependency_free(dep); | 465 | i915_dependency_free(dep); |
| 479 | 466 | ||
| 480 | local_bh_enable(); /* kick submission tasklet */ | 467 | local_bh_enable(); /* kick submission tasklet */ |
| @@ -34,7 +34,8 @@ bool __i915_sched_node_add_dependency(struct i915_sched_node *node, · 33 unmodified lines | |||
| 34 | unsigned long flags); | 34 | unsigned long flags); |
| 35 | 35 | ||
| 36 | int i915_sched_node_add_dependency(struct i915_sched_node *node, | 36 | int i915_sched_node_add_dependency(struct i915_sched_node *node, |
| 37 | struct i915_sched_node *signal); | 37 | struct i915_sched_node *signal, |
| 38 | unsigned long flags); | ||
| 38 | 39 | ||
| 39 | void i915_sched_node_fini(struct i915_sched_node *node); | 40 | void i915_sched_node_fini(struct i915_sched_node *node); |
| 40 | 41 | ||
| @@ -65,7 +65,7 @@ struct i915_sched_node { · 64 unmodified lines | |||
| 65 | struct list_head link; | 65 | struct list_head link; |
| 66 | struct i915_sched_attr attr; | 66 | struct i915_sched_attr attr; |
| 67 | unsigned int flags; | 67 | unsigned int flags; |
| 68 | #define I915_SCHED_HAS_SEMAPHORE_CHAIN BIT(0) | 68 | #define I915_SCHED_HAS_EXTERNAL_CHAIN BIT(0) |
| 69 | intel_engine_mask_t semaphores; | 69 | intel_engine_mask_t semaphores; |
| 70 | }; | 70 | }; |
| 71 | 71 | ||
| @@ -78,6 +78,7 @@ struct i915_dependency { · 6 unmodified lines | |||
| 78 | unsigned long flags; | 78 | unsigned long flags; |
| 79 | #define I915_DEPENDENCY_ALLOC BIT(0) | 79 | #define I915_DEPENDENCY_ALLOC BIT(0) |
| 80 | #define I915_DEPENDENCY_EXTERNAL BIT(1) | 80 | #define I915_DEPENDENCY_EXTERNAL BIT(1) |
| 81 | #define I915_DEPENDENCY_WEAK BIT(2) | ||
| 81 | }; | 82 | }; |
| 82 | 83 | ||
| 83 | #endif /* _I915_SCHEDULER_TYPES_H_ */ | 84 | #endif /* _I915_SCHEDULER_TYPES_H_ */ |
| @@ -133,4 +133,6 @@ bool __igt_timeout(unsigned long timeout, const char *fmt, ...); · 132 unmodified lines | |||
| 133 | #define igt_timeout(t, fmt, ...) \ | 133 | #define igt_timeout(t, fmt, ...) \ |
| 134 | __igt_timeout((t), KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__) | 134 | __igt_timeout((t), KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__) |
| 135 | 135 | ||
| 136 | void igt_hexdump(const void *buf, size_t len); | ||
| 137 | |||
| 136 | #endif /* !__I915_SELFTEST_H__ */ | 138 | #endif /* !__I915_SELFTEST_H__ */ |
| @@ -546,13 +546,11 @@ int __i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence, · 545 unmodified lines | |||
| 546 | cb->fence = fence; | 546 | cb->fence = fence; |
| 547 | i915_sw_fence_await(fence); | 547 | i915_sw_fence_await(fence); |
| 548 | 548 | ||
| 549 | ret = dma_fence_add_callback(dma, &cb->base, __dma_i915_sw_fence_wake); | 549 | ret = 1; |
| 550 | if (ret == 0) { | 550 | if (dma_fence_add_callback(dma, &cb->base, __dma_i915_sw_fence_wake)) { |
| 551 | ret = 1; | 551 | /* fence already signaled */ |
| 552 | } else { | ||
| 553 | __dma_i915_sw_fence_wake(dma, &cb->base); | 552 | __dma_i915_sw_fence_wake(dma, &cb->base); |
| 554 | if (ret == -ENOENT) /* fence already signaled */ | 553 | ret = 0; |
| 555 | ret = 0; | ||
| 556 | } | 554 | } |
| 557 | 555 | ||
| 558 | return ret; | 556 | return ret; |
| @@ -136,8 +136,8 @@ void intel_device_info_print_runtime(const struct intel_runtime_info *info, · 135 unmodified lines | |||
| 136 | sseu_dump(&info->sseu, p); | 136 | sseu_dump(&info->sseu, p); |
| 137 | 137 | ||
| 138 | drm_printf(p, "rawclk rate: %u kHz\n", info->rawclk_freq); | 138 | drm_printf(p, "rawclk rate: %u kHz\n", info->rawclk_freq); |
| 139 | drm_printf(p, "CS timestamp frequency: %u kHz\n", | 139 | drm_printf(p, "CS timestamp frequency: %u Hz\n", |
| 140 | info->cs_timestamp_frequency_khz); | 140 | info->cs_timestamp_frequency_hz); |
| 141 | } | 141 | } |
| 142 | 142 | ||
| 143 | static int sseu_eu_idx(const struct sseu_dev_info *sseu, int slice, | 143 | static int sseu_eu_idx(const struct sseu_dev_info *sseu, int slice, |
| @@ -678,12 +678,12 @@ static u32 read_reference_ts_freq(struct drm_i915_private *dev_priv) · 534 unmodified lines | |||
| 678 | 678 | ||
| 679 | base_freq = ((ts_override & GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK) >> | 679 | base_freq = ((ts_override & GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK) >> |
| 680 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_SHIFT) + 1; | 680 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_SHIFT) + 1; |
| 681 | base_freq *= 1000; | 681 | base_freq *= 1000000; |
| 682 | 682 | ||
| 683 | frac_freq = ((ts_override & | 683 | frac_freq = ((ts_override & |
| 684 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_MASK) >> | 684 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_MASK) >> |
| 685 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_SHIFT); | 685 | GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_SHIFT); |
| 686 | frac_freq = 1000 / (frac_freq + 1); | 686 | frac_freq = 1000000 / (frac_freq + 1); |
| 687 | 687 | ||
| 688 | return base_freq + frac_freq; | 688 | return base_freq + frac_freq; |
| 689 | } | 689 | } |
| @@ -691,8 +691,8 @@ static u32 read_reference_ts_freq(struct drm_i915_private *dev_priv) · 1 unmodified line | |||
| 691 | static u32 gen10_get_crystal_clock_freq(struct drm_i915_private *dev_priv, | 691 | static u32 gen10_get_crystal_clock_freq(struct drm_i915_private *dev_priv, |
| 692 | u32 rpm_config_reg) | 692 | u32 rpm_config_reg) |
| 693 | { | 693 | { |
| 694 | u32 f19_2_mhz = 19200; | 694 | u32 f19_2_mhz = 19200000; |
| 695 | u32 f24_mhz = 24000; | 695 | u32 f24_mhz = 24000000; |
| 696 | u32 crystal_clock = (rpm_config_reg & | 696 | u32 crystal_clock = (rpm_config_reg & |
| 697 | GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >> | 697 | GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >> |
| 698 | GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT; | 698 | GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT; |
| @@ -711,10 +711,10 @@ static u32 gen10_get_crystal_clock_freq(struct drm_i915_private *dev_priv, · 12 unmodified lines | |||
| 711 | static u32 gen11_get_crystal_clock_freq(struct drm_i915_private *dev_priv, | 711 | static u32 gen11_get_crystal_clock_freq(struct drm_i915_private *dev_priv, |
| 712 | u32 rpm_config_reg) | 712 | u32 rpm_config_reg) |
| 713 | { | 713 | { |
| 714 | u32 f19_2_mhz = 19200; | 714 | u32 f19_2_mhz = 19200000; |
| 715 | u32 f24_mhz = 24000; | 715 | u32 f24_mhz = 24000000; |
| 716 | u32 f25_mhz = 25000; | 716 | u32 f25_mhz = 25000000; |
| 717 | u32 f38_4_mhz = 38400; | 717 | u32 f38_4_mhz = 38400000; |
| 718 | u32 crystal_clock = (rpm_config_reg & | 718 | u32 crystal_clock = (rpm_config_reg & |
| 719 | GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >> | 719 | GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >> |
| 720 | GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT; | 720 | GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT; |
| @@ -736,9 +736,9 @@ static u32 gen11_get_crystal_clock_freq(struct drm_i915_private *dev_priv, · 15 unmodified lines | |||
| 736 | 736 | ||
| 737 | static u32 read_timestamp_frequency(struct drm_i915_private *dev_priv) | 737 | static u32 read_timestamp_frequency(struct drm_i915_private *dev_priv) |
| 738 | { | 738 | { |
| 739 | u32 f12_5_mhz = 12500; | 739 | u32 f12_5_mhz = 12500000; |
| 740 | u32 f19_2_mhz = 19200; | 740 | u32 f19_2_mhz = 19200000; |
| 741 | u32 f24_mhz = 24000; | 741 | u32 f24_mhz = 24000000; |
| 742 | 742 | ||
| 743 | if (INTEL_GEN(dev_priv) <= 4) { | 743 | if (INTEL_GEN(dev_priv) <= 4) { |
| 744 | /* PRMs say: | 744 | /* PRMs say: |
| @@ -747,7 +747,7 @@ static u32 read_timestamp_frequency(struct drm_i915_private *dev_priv) · 2 unmodified lines | |||
| 747 | * hclks." (through the “Clocking Configuration” | 747 | * hclks." (through the “Clocking Configuration” |
| 748 | * (“CLKCFG”) MCHBAR register) | 748 | * (“CLKCFG”) MCHBAR register) |
| 749 | */ | 749 | */ |
| 750 | return RUNTIME_INFO(dev_priv)->rawclk_freq / 16; | 750 | return RUNTIME_INFO(dev_priv)->rawclk_freq * 1000 / 16; |
| 751 | } else if (INTEL_GEN(dev_priv) <= 8) { | 751 | } else if (INTEL_GEN(dev_priv) <= 8) { |
| 752 | /* PRMs say: | 752 | /* PRMs say: |
| 753 | * | 753 | * |
| @@ -1048,11 +1048,11 @@ void intel_device_info_runtime_init(struct drm_i915_private *dev_priv) · 294 unmodified lines | |||
| 1048 | drm_dbg(&dev_priv->drm, "rawclk rate: %d kHz\n", runtime->rawclk_freq); | 1048 | drm_dbg(&dev_priv->drm, "rawclk rate: %d kHz\n", runtime->rawclk_freq); |
| 1049 | 1049 | ||
| 1050 | /* Initialize command stream timestamp frequency */ | 1050 | /* Initialize command stream timestamp frequency */ |
| 1051 | runtime->cs_timestamp_frequency_khz = | 1051 | runtime->cs_timestamp_frequency_hz = |
| 1052 | read_timestamp_frequency(dev_priv); | 1052 | read_timestamp_frequency(dev_priv); |
| 1053 | if (runtime->cs_timestamp_frequency_khz) { | 1053 | if (runtime->cs_timestamp_frequency_hz) { |
| 1054 | runtime->cs_timestamp_period_ns = | 1054 | runtime->cs_timestamp_period_ns = |
| 1055 | div_u64(1e6, runtime->cs_timestamp_frequency_khz); | 1055 | i915_cs_timestamp_ticks_to_ns(dev_priv, 1); |
| 1056 | drm_dbg(&dev_priv->drm, | 1056 | drm_dbg(&dev_priv->drm, |
| 1057 | "CS timestamp wraparound in %lldms\n", | 1057 | "CS timestamp wraparound in %lldms\n", |
| 1058 | div_u64(mul_u32_u32(runtime->cs_timestamp_period_ns, | 1058 | div_u64(mul_u32_u32(runtime->cs_timestamp_period_ns, |
| @@ -221,7 +221,7 @@ struct intel_runtime_info { · 220 unmodified lines | |||
| 221 | 221 | ||
| 222 | u32 rawclk_freq; | 222 | u32 rawclk_freq; |
| 223 | 223 | ||
| 224 | u32 cs_timestamp_frequency_khz; | 224 | u32 cs_timestamp_frequency_hz; |
| 225 | u32 cs_timestamp_period_ns; | 225 | u32 cs_timestamp_period_ns; |
| 226 | 226 | ||
| 227 | /* Media engine access to SFC per instance */ | 227 | /* Media engine access to SFC per instance */ |
| @@ -43,6 +43,7 @@ · 42 unmodified lines | |||
| 43 | #include "i915_fixed.h" | 43 | #include "i915_fixed.h" |
| 44 | #include "i915_irq.h" | 44 | #include "i915_irq.h" |
| 45 | #include "i915_trace.h" | 45 | #include "i915_trace.h" |
| 46 | #include "display/intel_bw.h" | ||
| 46 | #include "intel_pm.h" | 47 | #include "intel_pm.h" |
| 47 | #include "intel_sideband.h" | 48 | #include "intel_sideband.h" |
| 48 | #include "../../../platform/x86/intel_ips.h" | 49 | #include "../../../platform/x86/intel_ips.h" |
| @@ -3637,10 +3638,6 @@ static bool skl_needs_memory_bw_wa(struct drm_i915_private *dev_priv) · 3588 unmodified lines | |||
| 3637 | static bool | 3638 | static bool |
| 3638 | intel_has_sagv(struct drm_i915_private *dev_priv) | 3639 | intel_has_sagv(struct drm_i915_private *dev_priv) |
| 3639 | { | 3640 | { |
| 3640 | /* HACK! */ | ||
| 3641 | if (IS_GEN(dev_priv, 12)) | ||
| 3642 | return false; | ||
| 3643 | |||
| 3644 | return (IS_GEN9_BC(dev_priv) || INTEL_GEN(dev_priv) >= 10) && | 3641 | return (IS_GEN9_BC(dev_priv) || INTEL_GEN(dev_priv) >= 10) && |
| 3645 | dev_priv->sagv_status != I915_SAGV_NOT_CONTROLLED; | 3642 | dev_priv->sagv_status != I915_SAGV_NOT_CONTROLLED; |
| 3646 | } | 3643 | } |
| @@ -3760,34 +3757,116 @@ intel_disable_sagv(struct drm_i915_private *dev_priv) · 113 unmodified lines | |||
| 3760 | void intel_sagv_pre_plane_update(struct intel_atomic_state *state) | 3757 | void intel_sagv_pre_plane_update(struct intel_atomic_state *state) |
| 3761 | { | 3758 | { |
| 3762 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); | 3759 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); |
| 3760 | const struct intel_bw_state *new_bw_state; | ||
| 3761 | const struct intel_bw_state *old_bw_state; | ||
| 3762 | u32 new_mask = 0; | ||
| 3763 | |||
| 3764 | /* | ||
| 3765 | * Just return if we can't control SAGV or don't have it. | ||
| 3766 | * This is different from situation when we have SAGV but just can't | ||
| 3767 | * afford it due to DBuf limitation - in case if SAGV is completely | ||
| 3768 | * disabled in a BIOS, we are not even allowed to send a PCode request, | ||
| 3769 | * as it will throw an error. So have to check it here. | ||
| 3770 | */ | ||
| 3771 | if (!intel_has_sagv(dev_priv)) | ||
| 3772 | return; | ||
| 3773 | |||
| 3774 | new_bw_state = intel_atomic_get_new_bw_state(state); | ||
| 3775 | if (!new_bw_state) | ||
| 3776 | return; | ||
| 3763 | 3777 | ||
| 3764 | if (!intel_can_enable_sagv(state)) | 3778 | if (INTEL_GEN(dev_priv) < 11 && !intel_can_enable_sagv(dev_priv, new_bw_state)) { |
| 3765 | intel_disable_sagv(dev_priv); | 3779 | intel_disable_sagv(dev_priv); |
| 3780 | return; | ||
| 3781 | } | ||
| 3782 | |||
| 3783 | old_bw_state = intel_atomic_get_old_bw_state(state); | ||
| 3784 | /* | ||
| 3785 | * Nothing to mask | ||
| 3786 | */ | ||
| 3787 | if (new_bw_state->qgv_points_mask == old_bw_state->qgv_points_mask) | ||
| 3788 | return; | ||
| 3789 | |||
| 3790 | new_mask = old_bw_state->qgv_points_mask | new_bw_state->qgv_points_mask; | ||
| 3791 | |||
| 3792 | /* | ||
| 3793 | * If new mask is zero - means there is nothing to mask, | ||
| 3794 | * we can only unmask, which should be done in unmask. | ||
| 3795 | */ | ||
| 3796 | if (!new_mask) | ||
| 3797 | return; | ||
| 3798 | |||
| 3799 | /* | ||
| 3800 | * Restrict required qgv points before updating the configuration. | ||
| 3801 | * According to BSpec we can't mask and unmask qgv points at the same | ||
| 3802 | * time. Also masking should be done before updating the configuration | ||
| 3803 | * and unmasking afterwards. | ||
| 3804 | */ | ||
| 3805 | icl_pcode_restrict_qgv_points(dev_priv, new_mask); | ||
| 3766 | } | 3806 | } |
| 3767 | 3807 | ||
| 3768 | void intel_sagv_post_plane_update(struct intel_atomic_state *state) | 3808 | void intel_sagv_post_plane_update(struct intel_atomic_state *state) |
| 3769 | { | 3809 | { |
| 3770 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); | 3810 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); |
| 3811 | const struct intel_bw_state *new_bw_state; | ||
| 3812 | const struct intel_bw_state *old_bw_state; | ||
| 3813 | u32 new_mask = 0; | ||
| 3814 | |||
| 3815 | /* | ||
| 3816 | * Just return if we can't control SAGV or don't have it. | ||
| 3817 | * This is different from situation when we have SAGV but just can't | ||
| 3818 | * afford it due to DBuf limitation - in case if SAGV is completely | ||
| 3819 | * disabled in a BIOS, we are not even allowed to send a PCode request, | ||
| 3820 | * as it will throw an error. So have to check it here. | ||
| 3821 | */ | ||
| 3822 | if (!intel_has_sagv(dev_priv)) | ||
| 3823 | return; | ||
| 3824 | |||
| 3825 | new_bw_state = intel_atomic_get_new_bw_state(state); | ||
| 3826 | if (!new_bw_state) | ||
| 3827 | return; | ||
| 3771 | 3828 | ||
| 3772 | if (intel_can_enable_sagv(state)) | 3829 | if (INTEL_GEN(dev_priv) < 11 && intel_can_enable_sagv(dev_priv, new_bw_state)) { |
| 3773 | intel_enable_sagv(dev_priv); | 3830 | intel_enable_sagv(dev_priv); |
| 3831 | return; | ||
| 3832 | } | ||
| 3833 | |||
| 3834 | old_bw_state = intel_atomic_get_old_bw_state(state); | ||
| 3835 | /* | ||
| 3836 | * Nothing to unmask | ||
| 3837 | */ | ||
| 3838 | if (new_bw_state->qgv_points_mask == old_bw_state->qgv_points_mask) | ||
| 3839 | return; | ||
| 3840 | |||
| 3841 | new_mask = new_bw_state->qgv_points_mask; | ||
| 3842 | |||
| 3843 | /* | ||
| 3844 | * Allow required qgv points after updating the configuration. | ||
| 3845 | * According to BSpec we can't mask and unmask qgv points at the same | ||
| 3846 | * time. Also masking should be done before updating the configuration | ||
| 3847 | * and unmasking afterwards. | ||
| 3848 | */ | ||
| 3849 | icl_pcode_restrict_qgv_points(dev_priv, new_mask); | ||
| 3774 | } | 3850 | } |
| 3775 | 3851 | ||
| 3776 | static bool intel_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state) | 3852 | static bool skl_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state) |
| 3777 | { | 3853 | { |
| 3778 | struct drm_device *dev = crtc_state->uapi.crtc->dev; | ||
| 3779 | struct drm_i915_private *dev_priv = to_i915(dev); | ||
| 3780 | struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); | 3854 | struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); |
| 3855 | struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); | ||
| 3781 | struct intel_plane *plane; | 3856 | struct intel_plane *plane; |
| 3857 | const struct intel_plane_state *plane_state; | ||
| 3782 | int level, latency; | 3858 | int level, latency; |
| 3783 | 3859 | ||
| 3860 | if (!intel_has_sagv(dev_priv)) | ||
| 3861 | return false; | ||
| 3862 | |||
| 3784 | if (!crtc_state->hw.active) | 3863 | if (!crtc_state->hw.active) |
| 3785 | return true; | 3864 | return true; |
| 3786 | 3865 | ||
| 3787 | if (crtc_state->hw.adjusted_mode.flags & DRM_MODE_FLAG_INTERLACE) | 3866 | if (crtc_state->hw.adjusted_mode.flags & DRM_MODE_FLAG_INTERLACE) |
| 3788 | return false; | 3867 | return false; |
| 3789 | 3868 | ||
| 3790 | for_each_intel_plane_on_crtc(dev, crtc, plane) { | 3869 | intel_atomic_crtc_state_for_each_plane_state(plane, plane_state, crtc_state) { |
| 3791 | const struct skl_plane_wm *wm = | 3870 | const struct skl_plane_wm *wm = |
| 3792 | &crtc_state->wm.skl.optimal.planes[plane->id]; | 3871 | &crtc_state->wm.skl.optimal.planes[plane->id]; |
| 3793 | 3872 | ||
| @@ -3803,7 +3882,7 @@ static bool intel_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state · 9 unmodified lines | |||
| 3803 | latency = dev_priv->wm.skl_latency[level]; | 3882 | latency = dev_priv->wm.skl_latency[level]; |
| 3804 | 3883 | ||
| 3805 | if (skl_needs_memory_bw_wa(dev_priv) && | 3884 | if (skl_needs_memory_bw_wa(dev_priv) && |
| 3806 | plane->base.state->fb->modifier == | 3885 | plane_state->uapi.fb->modifier == |
| 3807 | I915_FORMAT_MOD_X_TILED) | 3886 | I915_FORMAT_MOD_X_TILED) |
| 3808 | latency += 15; | 3887 | latency += 15; |
| 3809 | 3888 | ||
| @@ -3819,35 +3898,110 @@ static bool intel_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state · 9 unmodified lines | |||
| 3819 | return true; | 3898 | return true; |
| 3820 | } | 3899 | } |
| 3821 | 3900 | ||
| 3822 | bool intel_can_enable_sagv(struct intel_atomic_state *state) | 3901 | static bool tgl_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state) |
| 3902 | { | ||
| 3903 | struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); | ||
| 3904 | enum plane_id plane_id; | ||
| 3905 | |||
| 3906 | if (!crtc_state->hw.active) | ||
| 3907 | return true; | ||
| 3908 | |||
| 3909 | for_each_plane_id_on_crtc(crtc, plane_id) { | ||
| 3910 | const struct skl_ddb_entry *plane_alloc = | ||
| 3911 | &crtc_state->wm.skl.plane_ddb_y[plane_id]; | ||
| 3912 | const struct skl_plane_wm *wm = | ||
| 3913 | &crtc_state->wm.skl.optimal.planes[plane_id]; | ||
| 3914 | |||
| 3915 | if (skl_ddb_entry_size(plane_alloc) < wm->sagv_wm0.min_ddb_alloc) | ||
| 3916 | return false; | ||
| 3917 | } | ||
| 3918 | |||
| 3919 | return true; | ||
| 3920 | } | ||
| 3921 | |||
| 3922 | static bool intel_crtc_can_enable_sagv(const struct intel_crtc_state *crtc_state) | ||
| 3923 | { | ||
| 3924 | struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); | ||
| 3925 | struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); | ||
| 3926 | |||
| 3927 | if (INTEL_GEN(dev_priv) >= 12) | ||
| 3928 | return tgl_crtc_can_enable_sagv(crtc_state); | ||
| 3929 | else | ||
| 3930 | return skl_crtc_can_enable_sagv(crtc_state); | ||
| 3931 | } | ||
| 3932 | |||
| 3933 | bool intel_can_enable_sagv(struct drm_i915_private *dev_priv, | ||
| 3934 | const struct intel_bw_state *bw_state) | ||
| 3935 | { | ||
| 3936 | if (INTEL_GEN(dev_priv) < 11 && | ||
| 3937 | bw_state->active_pipes && !is_power_of_2(bw_state->active_pipes)) | ||
| 3938 | return false; | ||
| 3939 | |||
| 3940 | return bw_state->pipe_sagv_reject == 0; | ||
| 3941 | } | ||
| 3942 | |||
| 3943 | static int intel_compute_sagv_mask(struct intel_atomic_state *state) | ||
| 3823 | { | 3944 | { |
| 3824 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); | 3945 | struct drm_i915_private *dev_priv = to_i915(state->base.dev); |
| 3946 | int ret; | ||
| 3825 | struct intel_crtc *crtc; | 3947 | struct intel_crtc *crtc; |
| 3826 | const struct intel_crtc_state *crtc_state; | 3948 | struct intel_crtc_state *new_crtc_state; |
| 3827 | enum pipe pipe; | 3949 | struct intel_bw_state *new_bw_state = NULL; |
| 3950 | const struct intel_bw_state *old_bw_state = NULL; | ||
| 3951 | int i; | ||
| 3828 | 3952 | ||
| 3829 | if (!intel_has_sagv(dev_priv)) | 3953 | for_each_new_intel_crtc_in_state(state, crtc, |
| 3830 | return false; | 3954 | new_crtc_state, i) { |
| 3955 | new_bw_state = intel_atomic_get_bw_state(state); | ||
| 3956 | if (IS_ERR(new_bw_state)) | ||
| 3957 | return PTR_ERR(new_bw_state); | ||
| 3831 | 3958 | ||
| 3832 | /* | 3959 | old_bw_state = intel_atomic_get_old_bw_state(state); |
| 3833 | * If there are no active CRTCs, no additional checks need be performed | ||
| 3834 | */ | ||
| 3835 | if (hweight8(state->active_pipes) == 0) | ||
| 3836 | return true; | ||
| 3837 | 3960 | ||
| 3838 | /* | 3961 | if (intel_crtc_can_enable_sagv(new_crtc_state)) |
| 3839 | * SKL+ workaround: bspec recommends we disable SAGV when we have | 3962 | new_bw_state->pipe_sagv_reject &= ~BIT(crtc->pipe); |
| 3840 | * more then one pipe enabled | 3963 | else |
| 3841 | */ | 3964 | new_bw_state->pipe_sagv_reject |= BIT(crtc->pipe); |
| 3842 | if (hweight8(state->active_pipes) > 1) | 3965 | } |
| 3843 | return false; | 3966 | |
| 3967 | if (!new_bw_state) | ||
| 3968 | return 0; | ||
| 3969 | |||
| 3970 | new_bw_state->active_pipes = | ||
| 3971 | intel_calc_active_pipes(state, old_bw_state->active_pipes); | ||
| 3972 | |||
| 3973 | if (new_bw_state->active_pipes != old_bw_state->active_pipes) { | ||
| 3974 | ret = intel_atomic_lock_global_state(&new_bw_state->base); | ||
| 3975 | if (ret) | ||
| 3976 | return ret; | ||
| 3977 | } | ||
| 3978 | |||
| 3979 | for_each_new_intel_crtc_in_state(state, crtc, | ||
| 3980 | new_crtc_state, i) { | ||
| 3981 | struct skl_pipe_wm *pipe_wm = &new_crtc_state->wm.skl.optimal; | ||
| 3844 | 3982 | ||
| 3845 | /* Since we're now guaranteed to only have one active CRTC... */ | 3983 | /* |
| 3846 | pipe = ffs(state->active_pipes) - 1; | 3984 | * We store use_sagv_wm in the crtc state rather than relying on |
| 3847 | crtc = intel_get_crtc_for_pipe(dev_priv, pipe); | 3985 | * that bw state since we have no convenient way to get at the |
| 3848 | crtc_state = to_intel_crtc_state(crtc->base.state); | 3986 | * latter from the plane commit hooks (especially in the legacy |
| 3987 | * cursor case) | ||
| 3988 | */ | ||
| 3989 | pipe_wm->use_sagv_wm = INTEL_GEN(dev_priv) >= 12 && | ||
| 3990 | intel_can_enable_sagv(dev_priv, new_bw_state); | ||
| 3991 | } | ||
| 3992 | |||
| 3993 | if (intel_can_enable_sagv(dev_priv, new_bw_state) != | ||
| 3994 | intel_can_enable_sagv(dev_priv, old_bw_state)) { | ||
| 3995 | ret = intel_atomic_serialize_global_state(&new_bw_state->base); | ||
| 3996 | if (ret) | ||
| 3997 | return ret; | ||
| 3998 | } else if (new_bw_state->pipe_sagv_reject != old_bw_state->pipe_sagv_reject) { | ||
| 3999 | ret = intel_atomic_lock_global_state(&new_bw_state->base); | ||
| 4000 | if (ret) | ||
| 4001 | return ret; | ||
| 4002 | } | ||
| 3849 | 4003 | ||
| 3850 | return intel_crtc_can_enable_sagv(crtc_state); | 4004 | return 0; |
| 3851 | } | 4005 | } |
| 3852 | 4006 | ||
| 3853 | /* | 4007 | /* |
| @@ -4574,6 +4728,20 @@ icl_get_total_relative_data_rate(struct intel_crtc_state *crtc_state, · 720 unmodified lines | |||
| 4574 | return total_data_rate; | 4728 | return total_data_rate; |
| 4575 | } | 4729 | } |
| 4576 | 4730 | ||
| 4731 | static const struct skl_wm_level * | ||
| 4732 | skl_plane_wm_level(const struct intel_crtc_state *crtc_state, | ||
| 4733 | enum plane_id plane_id, | ||
| 4734 | int level) | ||
| 4735 | { | ||
| 4736 | const struct skl_pipe_wm *pipe_wm = &crtc_state->wm.skl.optimal; | ||
| 4737 | const struct skl_plane_wm *wm = &pipe_wm->planes[plane_id]; | ||
| 4738 | |||
| 4739 | if (level == 0 && pipe_wm->use_sagv_wm) | ||
| 4740 | return &wm->sagv_wm0; | ||
| 4741 | |||
| 4742 | return &wm->wm[level]; | ||
| 4743 | } | ||
| 4744 | |||
| 4577 | static int | 4745 | static int |
| 4578 | skl_allocate_pipe_ddb(struct intel_crtc_state *crtc_state) | 4746 | skl_allocate_pipe_ddb(struct intel_crtc_state *crtc_state) |
| 4579 | { | 4747 | { |
| @@ -4610,7 +4778,6 @@ skl_allocate_pipe_ddb(struct intel_crtc_state *crtc_state) · 30 unmodified lines | |||
| 4610 | plane_data_rate, | 4778 | plane_data_rate, |
| 4611 | uv_plane_data_rate); | 4779 | uv_plane_data_rate); |
| 4612 | 4780 | ||
| 4613 | |||
| 4614 | skl_ddb_get_pipe_allocation_limits(dev_priv, crtc_state, total_data_rate, | 4781 | skl_ddb_get_pipe_allocation_limits(dev_priv, crtc_state, total_data_rate, |
| 4615 | alloc, &num_active); | 4782 | alloc, &num_active); |
| 4616 | alloc_size = skl_ddb_entry_size(alloc); | 4783 | alloc_size = skl_ddb_entry_size(alloc); |
| @@ -4810,7 +4977,7 @@ skl_wm_method1(const struct drm_i915_private *dev_priv, u32 pixel_rate, · 193 unmodified lines | |||
| 4810 | wm_intermediate_val = latency * pixel_rate * cpp; | 4977 | wm_intermediate_val = latency * pixel_rate * cpp; |
| 4811 | ret = div_fixed16(wm_intermediate_val, 1000 * dbuf_block_size); | 4978 | ret = div_fixed16(wm_intermediate_val, 1000 * dbuf_block_size); |
| 4812 | 4979 | ||
| 4813 | if (INTEL_GEN(dev_priv) >= 10) | 4980 | if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv)) |
| 4814 | ret = add_fixed16_u32(ret, 1); | 4981 | ret = add_fixed16_u32(ret, 1); |
| 4815 | 4982 | ||
| 4816 | return ret; | 4983 | return ret; |
| @@ -4945,18 +5112,19 @@ skl_compute_wm_params(const struct intel_crtc_state *crtc_state, · 128 unmodified lines | |||
| 4945 | wp->y_min_scanlines, | 5112 | wp->y_min_scanlines, |
| 4946 | wp->dbuf_block_size); | 5113 | wp->dbuf_block_size); |
| 4947 | 5114 | ||
| 4948 | if (INTEL_GEN(dev_priv) >= 10) | 5115 | if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv)) |
| 4949 | interm_pbpl++; | 5116 | interm_pbpl++; |
| 4950 | 5117 | ||
| 4951 | wp->plane_blocks_per_line = div_fixed16(interm_pbpl, | 5118 | wp->plane_blocks_per_line = div_fixed16(interm_pbpl, |
| 4952 | wp->y_min_scanlines); | 5119 | wp->y_min_scanlines); |
| 4953 | } else if (wp->x_tiled && IS_GEN(dev_priv, 9)) { | ||
| 4954 | interm_pbpl = DIV_ROUND_UP(wp->plane_bytes_per_line, | ||
| 4955 | wp->dbuf_block_size); | ||
| 4956 | wp->plane_blocks_per_line = u32_to_fixed16(interm_pbpl); | ||
| 4957 | } else { | 5120 | } else { |
| 4958 | interm_pbpl = DIV_ROUND_UP(wp->plane_bytes_per_line, | 5121 | interm_pbpl = DIV_ROUND_UP(wp->plane_bytes_per_line, |
| 4959 | wp->dbuf_block_size) + 1; | 5122 | wp->dbuf_block_size); |
| 5123 | |||
| 5124 | if (!wp->x_tiled || | ||
| 5125 | INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv)) | ||
| 5126 | interm_pbpl++; | ||
| 5127 | |||
| 4960 | wp->plane_blocks_per_line = u32_to_fixed16(interm_pbpl); | 5128 | wp->plane_blocks_per_line = u32_to_fixed16(interm_pbpl); |
| 4961 | } | 5129 | } |
| 4962 | 5130 | ||
| @@ -5022,7 +5190,7 @@ static void skl_compute_plane_wm(const struct intel_crtc_state *crtc_state, · 59 unmodified lines | |||
| 5022 | * WaIncreaseLatencyIPCEnabled: kbl,cfl | 5190 | * WaIncreaseLatencyIPCEnabled: kbl,cfl |
| 5023 | * Display WA #1141: kbl,cfl | 5191 | * Display WA #1141: kbl,cfl |
| 5024 | */ | 5192 | */ |
| 5025 | if ((IS_KABYLAKE(dev_priv) || IS_COFFEELAKE(dev_priv)) || | 5193 | if ((IS_KABYLAKE(dev_priv) || IS_COFFEELAKE(dev_priv)) && |
| 5026 | dev_priv->ipc_enabled) | 5194 | dev_priv->ipc_enabled) |
| 5027 | latency += 4; | 5195 | latency += 4; |
| 5028 | 5196 | ||
| @@ -5145,6 +5313,20 @@ skl_compute_wm_levels(const struct intel_crtc_state *crtc_state, · 116 unmodified lines | |||
| 5145 | } | 5313 | } |
| 5146 | } | 5314 | } |
| 5147 | 5315 | ||
| 5316 | static void tgl_compute_sagv_wm(const struct intel_crtc_state *crtc_state, | ||
| 5317 | const struct skl_wm_params *wm_params, | ||
| 5318 | struct skl_plane_wm *plane_wm) | ||
| 5319 | { | ||
| 5320 | struct drm_i915_private *dev_priv = to_i915(crtc_state->uapi.crtc->dev); | ||
| 5321 | struct skl_wm_level *sagv_wm = &plane_wm->sagv_wm0; | ||
| 5322 | struct skl_wm_level *levels = plane_wm->wm; | ||
| 5323 | unsigned int latency = dev_priv->wm.skl_latency[0] + dev_priv->sagv_block_time_us; | ||
| 5324 | |||
| 5325 | skl_compute_plane_wm(crtc_state, 0, latency, | ||
| 5326 | wm_params, &levels[0], | ||
| 5327 | sagv_wm); | ||
| 5328 | } | ||
| 5329 | |||
| 5148 | static void skl_compute_transition_wm(const struct intel_crtc_state *crtc_state, | 5330 | static void skl_compute_transition_wm(const struct intel_crtc_state *crtc_state, |
| 5149 | const struct skl_wm_params *wp, | 5331 | const struct skl_wm_params *wp, |
| 5150 | struct skl_plane_wm *wm) | 5332 | struct skl_plane_wm *wm) |
| @@ -5197,10 +5379,6 @@ static void skl_compute_transition_wm(const struct intel_crtc_state *crtc_state, · 46 unmodified lines | |||
| 5197 | trans_offset_b; | 5379 | trans_offset_b; |
| 5198 | } else { | 5380 | } else { |
| 5199 | res_blocks = wm0_sel_res_b + trans_offset_b; | 5381 | res_blocks = wm0_sel_res_b + trans_offset_b; |
| 5200 | |||
| 5201 | /* WA BUG:1938466 add one block for non y-tile planes */ | ||
| 5202 | if (IS_CNL_REVID(dev_priv, CNL_REVID_A0, CNL_REVID_A0)) | ||
| 5203 | res_blocks += 1; | ||
| 5204 | } | 5382 | } |
| 5205 | 5383 | ||
| 5206 | /* | 5384 | /* |
| @@ -5216,6 +5394,8 @@ static int skl_build_plane_wm_single(struct intel_crtc_state *crtc_state, · 9 unmodified lines | |||
| 5216 | const struct intel_plane_state *plane_state, | 5394 | const struct intel_plane_state *plane_state, |
| 5217 | enum plane_id plane_id, int color_plane) | 5395 | enum plane_id plane_id, int color_plane) |
| 5218 | { | 5396 | { |
| 5397 | struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc); | ||
| 5398 | struct drm_i915_private *dev_priv = to_i915(crtc->base.dev); | ||
| 5219 | struct skl_plane_wm *wm = &crtc_state->wm.skl.optimal.planes[plane_id]; | 5399 | struct skl_plane_wm *wm = &crtc_state->wm.skl.optimal.planes[plane_id]; |
| 5220 | struct skl_wm_params wm_params; | 5400 | struct skl_wm_params wm_params; |
| 5221 | int ret; | 5401 | int ret; |
| @@ -5226,6 +5406,10 @@ static int skl_build_plane_wm_single(struct intel_crtc_state *crtc_state, · 4 unmodified lines | |||
| 5226 | return ret; | 5406 | return ret; |
| 5227 | 5407 | ||
| 5228 | skl_compute_wm_levels(crtc_state, &wm_params, wm->wm); | 5408 | skl_compute_wm_levels(crtc_state, &wm_params, wm->wm); |
| 5409 | |||
| 5410 | if (INTEL_GEN(dev_priv) >= 12) | ||
| 5411 | tgl_compute_sagv_wm(crtc_state, &wm_params, wm); | ||
| 5412 | |||
| 5229 | skl_compute_transition_wm(crtc_state, &wm_params, wm); | 5413 | skl_compute_transition_wm(crtc_state, &wm_params, wm); |
| 5230 | 5414 | ||
| 5231 | return 0; | 5415 | return 0; |
| @@ -5385,8 +5569,12 @@ void skl_write_plane_wm(struct intel_plane *plane, · 153 unmodified lines | |||
| 5385 | &crtc_state->wm.skl.plane_ddb_uv[plane_id]; | 5569 | &crtc_state->wm.skl.plane_ddb_uv[plane_id]; |
| 5386 | 5570 | ||
| 5387 | for (level = 0; level <= max_level; level++) { | 5571 | for (level = 0; level <= max_level; level++) { |
| 5572 | const struct skl_wm_level *wm_level; | ||
| 5573 | |||
| 5574 | wm_level = skl_plane_wm_level(crtc_state, plane_id, level); | ||
| 5575 | |||
| 5388 | skl_write_wm_level(dev_priv, PLANE_WM(pipe, plane_id, level), | 5576 | skl_write_wm_level(dev_priv, PLANE_WM(pipe, plane_id, level), |
| 5389 | &wm->wm[level]); | 5577 | wm_level); |
| 5390 | } | 5578 | } |
| 5391 | skl_write_wm_level(dev_priv, PLANE_WM_TRANS(pipe, plane_id), | 5579 | skl_write_wm_level(dev_priv, PLANE_WM_TRANS(pipe, plane_id), |
| 5392 | &wm->trans_wm); | 5580 | &wm->trans_wm); |
| @@ -5419,8 +5607,12 @@ void skl_write_cursor_wm(struct intel_plane *plane, · 26 unmodified lines | |||
| 5419 | &crtc_state->wm.skl.plane_ddb_y[plane_id]; | 5607 | &crtc_state->wm.skl.plane_ddb_y[plane_id]; |
| 5420 | 5608 | ||
| 5421 | for (level = 0; level <= max_level; level++) { | 5609 | for (level = 0; level <= max_level; level++) { |
| 5610 | const struct skl_wm_level *wm_level; | ||
| 5611 | |||
| 5612 | wm_level = skl_plane_wm_level(crtc_state, plane_id, level); | ||
| 5613 | |||
| 5422 | skl_write_wm_level(dev_priv, CUR_WM(pipe, level), | 5614 | skl_write_wm_level(dev_priv, CUR_WM(pipe, level), |
| 5423 | &wm->wm[level]); | 5615 | wm_level); |
| 5424 | } | 5616 | } |
| 5425 | skl_write_wm_level(dev_priv, CUR_WM_TRANS(pipe), &wm->trans_wm); | 5617 | skl_write_wm_level(dev_priv, CUR_WM_TRANS(pipe), &wm->trans_wm); |
| 5426 | 5618 | ||
| @@ -5584,23 +5776,25 @@ skl_print_wm_changes(struct intel_atomic_state *state) · 157 unmodified lines | |||
| 5584 | continue; | 5776 | continue; |
| 5585 | 5777 | ||
| 5586 | drm_dbg_kms(&dev_priv->drm, | 5778 | drm_dbg_kms(&dev_priv->drm, |
| 5587 | "[PLANE:%d:%s] level %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm" | 5779 | "[PLANE:%d:%s] level %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm,%cswm" |
| 5588 | " -> %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm\n", | 5780 | " -> %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm,%cswm\n", |
| 5589 | plane->base.base.id, plane->base.name, | 5781 | plane->base.base.id, plane->base.name, |
| 5590 | enast(old_wm->wm[0].plane_en), enast(old_wm->wm[1].plane_en), | 5782 | enast(old_wm->wm[0].plane_en), enast(old_wm->wm[1].plane_en), |
| 5591 | enast(old_wm->wm[2].plane_en), enast(old_wm->wm[3].plane_en), | 5783 | enast(old_wm->wm[2].plane_en), enast(old_wm->wm[3].plane_en), |
| 5592 | enast(old_wm->wm[4].plane_en), enast(old_wm->wm[5].plane_en), | 5784 | enast(old_wm->wm[4].plane_en), enast(old_wm->wm[5].plane_en), |
| 5593 | enast(old_wm->wm[6].plane_en), enast(old_wm->wm[7].plane_en), | 5785 | enast(old_wm->wm[6].plane_en), enast(old_wm->wm[7].plane_en), |
| 5594 | enast(old_wm->trans_wm.plane_en), | 5786 | enast(old_wm->trans_wm.plane_en), |
| 5787 | enast(old_wm->sagv_wm0.plane_en), | ||
| 5595 | enast(new_wm->wm[0].plane_en), enast(new_wm->wm[1].plane_en), | 5788 | enast(new_wm->wm[0].plane_en), enast(new_wm->wm[1].plane_en), |
| 5596 | enast(new_wm->wm[2].plane_en), enast(new_wm->wm[3].plane_en), | 5789 | enast(new_wm->wm[2].plane_en), enast(new_wm->wm[3].plane_en), |
| 5597 | enast(new_wm->wm[4].plane_en), enast(new_wm->wm[5].plane_en), | 5790 | enast(new_wm->wm[4].plane_en), enast(new_wm->wm[5].plane_en), |
| 5598 | enast(new_wm->wm[6].plane_en), enast(new_wm->wm[7].plane_en), | 5791 | enast(new_wm->wm[6].plane_en), enast(new_wm->wm[7].plane_en), |
| 5599 | enast(new_wm->trans_wm.plane_en)); | 5792 | enast(new_wm->trans_wm.plane_en), |
| 5793 | enast(new_wm->sagv_wm0.plane_en)); | ||
| 5600 | 5794 | ||
| 5601 | drm_dbg_kms(&dev_priv->drm, | 5795 | drm_dbg_kms(&dev_priv->drm, |
| 5602 | "[PLANE:%d:%s] lines %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d" | 5796 | "[PLANE:%d:%s] lines %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d" |
| 5603 | " -> %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d\n", | 5797 | " -> %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d\n", |
| 5604 | plane->base.base.id, plane->base.name, | 5798 | plane->base.base.id, plane->base.name, |
| 5605 | enast(old_wm->wm[0].ignore_lines), old_wm->wm[0].plane_res_l, | 5799 | enast(old_wm->wm[0].ignore_lines), old_wm->wm[0].plane_res_l, |
| 5606 | enast(old_wm->wm[1].ignore_lines), old_wm->wm[1].plane_res_l, | 5800 | enast(old_wm->wm[1].ignore_lines), old_wm->wm[1].plane_res_l, |
| @@ -5611,6 +5805,7 @@ skl_print_wm_changes(struct intel_atomic_state *state) · 4 unmodified lines | |||
| 5611 | enast(old_wm->wm[6].ignore_lines), old_wm->wm[6].plane_res_l, | 5805 | enast(old_wm->wm[6].ignore_lines), old_wm->wm[6].plane_res_l, |
| 5612 | enast(old_wm->wm[7].ignore_lines), old_wm->wm[7].plane_res_l, | 5806 | enast(old_wm->wm[7].ignore_lines), old_wm->wm[7].plane_res_l, |
| 5613 | enast(old_wm->trans_wm.ignore_lines), old_wm->trans_wm.plane_res_l, | 5807 | enast(old_wm->trans_wm.ignore_lines), old_wm->trans_wm.plane_res_l, |
| 5808 | enast(old_wm->sagv_wm0.ignore_lines), old_wm->sagv_wm0.plane_res_l, | ||
| 5614 | 5809 | ||
| 5615 | enast(new_wm->wm[0].ignore_lines), new_wm->wm[0].plane_res_l, | 5810 | enast(new_wm->wm[0].ignore_lines), new_wm->wm[0].plane_res_l, |
| 5616 | enast(new_wm->wm[1].ignore_lines), new_wm->wm[1].plane_res_l, | 5811 | enast(new_wm->wm[1].ignore_lines), new_wm->wm[1].plane_res_l, |
| @@ -5620,37 +5815,42 @@ skl_print_wm_changes(struct intel_atomic_state *state) · 3 unmodified lines | |||
| 5620 | enast(new_wm->wm[5].ignore_lines), new_wm->wm[5].plane_res_l, | 5815 | enast(new_wm->wm[5].ignore_lines), new_wm->wm[5].plane_res_l, |
| 5621 | enast(new_wm->wm[6].ignore_lines), new_wm->wm[6].plane_res_l, | 5816 | enast(new_wm->wm[6].ignore_lines), new_wm->wm[6].plane_res_l, |
| 5622 | enast(new_wm->wm[7].ignore_lines), new_wm->wm[7].plane_res_l, | 5817 | enast(new_wm->wm[7].ignore_lines), new_wm->wm[7].plane_res_l, |
| 5623 | enast(new_wm->trans_wm.ignore_lines), new_wm->trans_wm.plane_res_l); | 5818 | enast(new_wm->trans_wm.ignore_lines), new_wm->trans_wm.plane_res_l, |
| 5819 | enast(new_wm->sagv_wm0.ignore_lines), new_wm->sagv_wm0.plane_res_l); | ||
| 5624 | 5820 | ||
| 5625 | drm_dbg_kms(&dev_priv->drm, | 5821 | drm_dbg_kms(&dev_priv->drm, |
| 5626 | "[PLANE:%d:%s] blocks %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d" | 5822 | "[PLANE:%d:%s] blocks %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d" |
| 5627 | " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n", | 5823 | " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n", |
| 5628 | plane->base.base.id, plane->base.name, | 5824 | plane->base.base.id, plane->base.name, |
| 5629 | old_wm->wm[0].plane_res_b, old_wm->wm[1].plane_res_b, | 5825 | old_wm->wm[0].plane_res_b, old_wm->wm[1].plane_res_b, |
| 5630 | old_wm->wm[2].plane_res_b, old_wm->wm[3].plane_res_b, | 5826 | old_wm->wm[2].plane_res_b, old_wm->wm[3].plane_res_b, |
| 5631 | old_wm->wm[4].plane_res_b, old_wm->wm[5].plane_res_b, | 5827 | old_wm->wm[4].plane_res_b, old_wm->wm[5].plane_res_b, |
| 5632 | old_wm->wm[6].plane_res_b, old_wm->wm[7].plane_res_b, | 5828 | old_wm->wm[6].plane_res_b, old_wm->wm[7].plane_res_b, |
| 5633 | old_wm->trans_wm.plane_res_b, | 5829 | old_wm->trans_wm.plane_res_b, |
| 5830 | old_wm->sagv_wm0.plane_res_b, | ||
| 5634 | new_wm->wm[0].plane_res_b, new_wm->wm[1].plane_res_b, | 5831 | new_wm->wm[0].plane_res_b, new_wm->wm[1].plane_res_b, |
| 5635 | new_wm->wm[2].plane_res_b, new_wm->wm[3].plane_res_b, | 5832 | new_wm->wm[2].plane_res_b, new_wm->wm[3].plane_res_b, |
| 5636 | new_wm->wm[4].plane_res_b, new_wm->wm[5].plane_res_b, | 5833 | new_wm->wm[4].plane_res_b, new_wm->wm[5].plane_res_b, |
| 5637 | new_wm->wm[6].plane_res_b, new_wm->wm[7].plane_res_b, | 5834 | new_wm->wm[6].plane_res_b, new_wm->wm[7].plane_res_b, |
| 5638 | new_wm->trans_wm.plane_res_b); | 5835 | new_wm->trans_wm.plane_res_b, |
| 5836 | new_wm->sagv_wm0.plane_res_b); | ||
| 5639 | 5837 | ||
| 5640 | drm_dbg_kms(&dev_priv->drm, | 5838 | drm_dbg_kms(&dev_priv->drm, |
| 5641 | "[PLANE:%d:%s] min_ddb %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d" | 5839 | "[PLANE:%d:%s] min_ddb %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d" |
| 5642 | " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n", | 5840 | " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n", |
| 5643 | plane->base.base.id, plane->base.name, | 5841 | plane->base.base.id, plane->base.name, |
| 5644 | old_wm->wm[0].min_ddb_alloc, old_wm->wm[1].min_ddb_alloc, | 5842 | old_wm->wm[0].min_ddb_alloc, old_wm->wm[1].min_ddb_alloc, |
| 5645 | old_wm->wm[2].min_ddb_alloc, old_wm->wm[3].min_ddb_alloc, | 5843 | old_wm->wm[2].min_ddb_alloc, old_wm->wm[3].min_ddb_alloc, |
| 5646 | old_wm->wm[4].min_ddb_alloc, old_wm->wm[5].min_ddb_alloc, | 5844 | old_wm->wm[4].min_ddb_alloc, old_wm->wm[5].min_ddb_alloc, |
| 5647 | old_wm->wm[6].min_ddb_alloc, old_wm->wm[7].min_ddb_alloc, | 5845 | old_wm->wm[6].min_ddb_alloc, old_wm->wm[7].min_ddb_alloc, |
| 5648 | old_wm->trans_wm.min_ddb_alloc, | 5846 | old_wm->trans_wm.min_ddb_alloc, |
| 5847 | old_wm->sagv_wm0.min_ddb_alloc, | ||
| 5649 | new_wm->wm[0].min_ddb_alloc, new_wm->wm[1].min_ddb_alloc, | 5848 | new_wm->wm[0].min_ddb_alloc, new_wm->wm[1].min_ddb_alloc, |
| 5650 | new_wm->wm[2].min_ddb_alloc, new_wm->wm[3].min_ddb_alloc, | 5849 | new_wm->wm[2].min_ddb_alloc, new_wm->wm[3].min_ddb_alloc, |
| 5651 | new_wm->wm[4].min_ddb_alloc, new_wm->wm[5].min_ddb_alloc, | 5850 | new_wm->wm[4].min_ddb_alloc, new_wm->wm[5].min_ddb_alloc, |
| 5652 | new_wm->wm[6].min_ddb_alloc, new_wm->wm[7].min_ddb_alloc, | 5851 | new_wm->wm[6].min_ddb_alloc, new_wm->wm[7].min_ddb_alloc, |
| 5653 | new_wm->trans_wm.min_ddb_alloc); | 5852 | new_wm->trans_wm.min_ddb_alloc, |
| 5853 | new_wm->sagv_wm0.min_ddb_alloc); | ||
| 5654 | } | 5854 | } |
| 5655 | } | 5855 | } |
| 5656 | } | 5856 | } |
| @@ -5811,6 +6011,10 @@ skl_compute_wm(struct intel_atomic_state *state) · 154 unmodified lines | |||
| 5811 | if (ret) | 6011 | if (ret) |
| 5812 | return ret; | 6012 | return ret; |
| 5813 | 6013 | ||
| 6014 | ret = intel_compute_sagv_mask(state); | ||
| 6015 | if (ret) | ||
| 6016 | return ret; | ||
| 6017 | |||
| 5814 | /* | 6018 | /* |
| 5815 | * skl_compute_ddb() will have adjusted the final watermarks | 6019 | * skl_compute_ddb() will have adjusted the final watermarks |
| 5816 | * based on how much ddb is available. Now we can actually | 6020 | * based on how much ddb is available. Now we can actually |
| @@ -5939,6 +6143,9 @@ void skl_pipe_wm_get_hw_state(struct intel_crtc *crtc, · 122 unmodified lines | |||
| 5939 | skl_wm_level_from_reg_val(val, &wm->wm[level]); | 6143 | skl_wm_level_from_reg_val(val, &wm->wm[level]); |
| 5940 | } | 6144 | } |
| 5941 | 6145 | ||
| 6146 | if (INTEL_GEN(dev_priv) >= 12) | ||
| 6147 | wm->sagv_wm0 = wm->wm[0]; | ||
| 6148 | |||
| 5942 | if (plane_id != PLANE_CURSOR) | 6149 | if (plane_id != PLANE_CURSOR) |
| 5943 | val = I915_READ(PLANE_WM_TRANS(pipe, plane_id)); | 6150 | val = I915_READ(PLANE_WM_TRANS(pipe, plane_id)); |
| 5944 | else | 6151 | else |
| @@ -6916,9 +7123,6 @@ static void cnl_init_clock_gating(struct drm_i915_private *dev_priv) · 971 unmodified lines | |||
| 6916 | val = I915_READ(SLICE_UNIT_LEVEL_CLKGATE); | 7123 | val = I915_READ(SLICE_UNIT_LEVEL_CLKGATE); |
| 6917 | /* ReadHitWriteOnlyDisable:cnl */ | 7124 | /* ReadHitWriteOnlyDisable:cnl */ |
| 6918 | val |= RCCUNIT_CLKGATE_DIS; | 7125 | val |= RCCUNIT_CLKGATE_DIS; |
| 6919 | /* WaSarbUnitClockGatingDisable:cnl (pre-prod) */ | ||
| 6920 | if (IS_CNL_REVID(dev_priv, CNL_REVID_A0, CNL_REVID_B0)) | ||
| 6921 | val |= SARBUNIT_CLKGATE_DIS; | ||
| 6922 | I915_WRITE(SLICE_UNIT_LEVEL_CLKGATE, val); | 7126 | I915_WRITE(SLICE_UNIT_LEVEL_CLKGATE, val); |
| 6923 | 7127 | ||
| 6924 | /* Wa_2201832410:cnl */ | 7128 | /* Wa_2201832410:cnl */ |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | #include <linux/types.h> | 9 | #include <linux/types.h> |
| 10 | 10 | ||
| 11 | #include "i915_reg.h" | 11 | #include "i915_reg.h" |
| 12 | #include "display/intel_bw.h" | ||
| 12 | 13 | ||
| 13 | struct drm_device; | 14 | struct drm_device; |
| 14 | struct drm_i915_private; | 15 | struct drm_i915_private; |
| @@ -41,7 +42,8 @@ void skl_pipe_wm_get_hw_state(struct intel_crtc *crtc, · 26 unmodified lines | |||
| 41 | struct skl_pipe_wm *out); | 42 | struct skl_pipe_wm *out); |
| 42 | void g4x_wm_sanitize(struct drm_i915_private *dev_priv); | 43 | void g4x_wm_sanitize(struct drm_i915_private *dev_priv); |
| 43 | void vlv_wm_sanitize(struct drm_i915_private *dev_priv); | 44 | void vlv_wm_sanitize(struct drm_i915_private *dev_priv); |
| 44 | bool intel_can_enable_sagv(struct intel_atomic_state *state); | 45 | bool intel_can_enable_sagv(struct drm_i915_private *dev_priv, |
| 46 | const struct intel_bw_state *bw_state); | ||
| 45 | int intel_enable_sagv(struct drm_i915_private *dev_priv); | 47 | int intel_enable_sagv(struct drm_i915_private *dev_priv); |
| 46 | int intel_disable_sagv(struct drm_i915_private *dev_priv); | 48 | int intel_disable_sagv(struct drm_i915_private *dev_priv); |
| 47 | void intel_sagv_pre_plane_update(struct intel_atomic_state *state); | 49 | void intel_sagv_pre_plane_update(struct intel_atomic_state *state); |
| @@ -371,6 +371,8 @@ static int gen7_check_mailbox_status(u32 mbox) · 370 unmodified lines | |||
| 371 | return -ENXIO; | 371 | return -ENXIO; |
| 372 | case GEN11_PCODE_LOCKED: | 372 | case GEN11_PCODE_LOCKED: |
| 373 | return -EBUSY; | 373 | return -EBUSY; |
| 374 | case GEN11_PCODE_REJECTED: | ||
| 375 | return -EACCES; | ||
| 374 | case GEN7_PCODE_MIN_FREQ_TABLE_GT_RATIO_OUT_OF_RANGE: | 376 | case GEN7_PCODE_MIN_FREQ_TABLE_GT_RATIO_OUT_OF_RANGE: |
| 375 | return -EOVERFLOW; | 377 | return -EOVERFLOW; |
| 376 | default: | 378 | default: |
| @@ -429,7 +431,7 @@ int sandybridge_pcode_read(struct drm_i915_private *i915, u32 mbox, · 52 unmodified lines | |||
| 429 | 431 | ||
| 430 | mutex_lock(&i915->sb_lock); | 432 | mutex_lock(&i915->sb_lock); |
| 431 | err = __sandybridge_pcode_rw(i915, mbox, val, val1, | 433 | err = __sandybridge_pcode_rw(i915, mbox, val, val1, |
| 432 | 500, 0, | 434 | 500, 20, |
| 433 | true); | 435 | true); |
| 434 | mutex_unlock(&i915->sb_lock); | 436 | mutex_unlock(&i915->sb_lock); |
| 435 | 437 | ||
| @@ -899,11 +899,6 @@ find_fw_domain(struct intel_uncore *uncore, u32 offset) · 898 unmodified lines | |||
| 899 | #define GEN_FW_RANGE(s, e, d) \ | 899 | #define GEN_FW_RANGE(s, e, d) \ |
| 900 | { .start = (s), .end = (e), .domains = (d) } | 900 | { .start = (s), .end = (e), .domains = (d) } |
| 901 | 901 | ||
| 902 | #define HAS_FWTABLE(dev_priv) \ | ||
| 903 | (INTEL_GEN(dev_priv) >= 9 || \ | ||
| 904 | IS_CHERRYVIEW(dev_priv) || \ | ||
| 905 | IS_VALLEYVIEW(dev_priv)) | ||
| 906 | |||
| 907 | /* *Must* be sorted by offset ranges! See intel_fw_table_check(). */ | 902 | /* *Must* be sorted by offset ranges! See intel_fw_table_check(). */ |
| 908 | static const struct intel_forcewake_range __vlv_fw_ranges[] = { | 903 | static const struct intel_forcewake_range __vlv_fw_ranges[] = { |
| 909 | GEN_FW_RANGE(0x2000, 0x3fff, FORCEWAKE_RENDER), | 904 | GEN_FW_RANGE(0x2000, 0x3fff, FORCEWAKE_RENDER), |
| @@ -149,8 +149,7 @@ static bool check_hw_restrictions(struct drm_i915_private *i915, · 148 unmodified lines | |||
| 149 | guc_wopcm_size)) | 149 | guc_wopcm_size)) |
| 150 | return false; | 150 | return false; |
| 151 | 151 | ||
| 152 | if ((IS_GEN(i915, 9) || | 152 | if (IS_GEN(i915, 9) && |
| 153 | IS_CNL_REVID(i915, CNL_REVID_A0, CNL_REVID_A0)) && | ||
| 154 | !gen9_check_huc_fw_fits(i915, guc_wopcm_size, huc_fw_size)) | 153 | !gen9_check_huc_fw_fits(i915, guc_wopcm_size, huc_fw_size)) |
| 155 | return false; | 154 | return false; |
| 156 | 155 | ||
| @@ -37,6 +37,7 @@ selftest(gem, i915_gem_live_selftests) · 36 unmodified lines | |||
| 37 | selftest(evict, i915_gem_evict_live_selftests) | 37 | selftest(evict, i915_gem_evict_live_selftests) |
| 38 | selftest(hugepages, i915_gem_huge_page_live_selftests) | 38 | selftest(hugepages, i915_gem_huge_page_live_selftests) |
| 39 | selftest(gem_contexts, i915_gem_context_live_selftests) | 39 | selftest(gem_contexts, i915_gem_context_live_selftests) |
| 40 | selftest(gem_execbuf, i915_gem_execbuffer_live_selftests) | ||
| 40 | selftest(blt, i915_gem_object_blt_live_selftests) | 41 | selftest(blt, i915_gem_object_blt_live_selftests) |
| 41 | selftest(client, i915_gem_client_blt_live_selftests) | 42 | selftest(client, i915_gem_client_blt_live_selftests) |
| 42 | selftest(reset, intel_reset_live_selftests) | 43 | selftest(reset, intel_reset_live_selftests) |
| @@ -221,8 +221,7 @@ static int live_noa_delay(void *arg) · 220 unmodified lines | |||
| 221 | goto out; | 221 | goto out; |
| 222 | } | 222 | } |
| 223 | 223 | ||
| 224 | if (rq->engine->emit_init_breadcrumb && | 224 | if (rq->engine->emit_init_breadcrumb) { |
| 225 | i915_request_timeline(rq)->has_initial_breadcrumb) { | ||
| 226 | err = rq->engine->emit_init_breadcrumb(rq); | 225 | err = rq->engine->emit_init_breadcrumb(rq); |
| 227 | if (err) { | 226 | if (err) { |
| 228 | i915_request_add(rq); | 227 | i915_request_add(rq); |
| @@ -263,8 +262,7 @@ static int live_noa_delay(void *arg) · 34 unmodified lines | |||
| 263 | 262 | ||
| 264 | delay = intel_read_status_page(stream->engine, 0x102); | 263 | delay = intel_read_status_page(stream->engine, 0x102); |
| 265 | delay -= intel_read_status_page(stream->engine, 0x100); | 264 | delay -= intel_read_status_page(stream->engine, 0x100); |
| 266 | delay = div_u64(mul_u32_u32(delay, 1000 * 1000), | 265 | delay = i915_cs_timestamp_ticks_to_ns(i915, delay); |
| 267 | RUNTIME_INFO(i915)->cs_timestamp_frequency_khz); | ||
| 268 | pr_info("GPU delay: %uns, expected %lluns\n", | 266 | pr_info("GPU delay: %uns, expected %lluns\n", |
| 269 | delay, expected); | 267 | delay, expected); |
| 270 | 268 | ||
| @@ -816,10 +816,12 @@ static int recursive_batch_resolve(struct i915_vma *batch) · 815 unmodified lines | |||
| 816 | return PTR_ERR(cmd); | 816 | return PTR_ERR(cmd); |
| 817 | 817 | ||
| 818 | *cmd = MI_BATCH_BUFFER_END; | 818 | *cmd = MI_BATCH_BUFFER_END; |
| 819 | intel_gt_chipset_flush(batch->vm->gt); | ||
| 820 | 819 | ||
| 820 | __i915_gem_object_flush_map(batch->obj, 0, sizeof(*cmd)); | ||
| 821 | i915_gem_object_unpin_map(batch->obj); | 821 | i915_gem_object_unpin_map(batch->obj); |
| 822 | 822 | ||
| 823 | intel_gt_chipset_flush(batch->vm->gt); | ||
| 824 | |||
| 823 | return 0; | 825 | return 0; |
| 824 | } | 826 | } |
| 825 | 827 | ||
| @@ -865,13 +867,6 @@ static int live_all_engines(void *arg) · 39 unmodified lines | |||
| 865 | goto out_request; | 867 | goto out_request; |
| 866 | } | 868 | } |
| 867 | 869 | ||
| 868 | err = engine->emit_bb_start(request[idx], | ||
| 869 | batch->node.start, | ||
| 870 | batch->node.size, | ||
| 871 | 0); | ||
| 872 | GEM_BUG_ON(err); | ||
| 873 | request[idx]->batch = batch; | ||
| 874 | |||
| 875 | i915_vma_lock(batch); | 870 | i915_vma_lock(batch); |
| 876 | err = i915_request_await_object(request[idx], batch->obj, 0); | 871 | err = i915_request_await_object(request[idx], batch->obj, 0); |
| 877 | if (err == 0) | 872 | if (err == 0) |
| @@ -879,6 +874,13 @@ static int live_all_engines(void *arg) · 1 unmodified line | |||
| 879 | i915_vma_unlock(batch); | 874 | i915_vma_unlock(batch); |
| 880 | GEM_BUG_ON(err); | 875 | GEM_BUG_ON(err); |
| 881 | 876 | ||
| 877 | err = engine->emit_bb_start(request[idx], | ||
| 878 | batch->node.start, | ||
| 879 | batch->node.size, | ||
| 880 | 0); | ||
| 881 | GEM_BUG_ON(err); | ||
| 882 | request[idx]->batch = batch; | ||
| 883 | |||
| 882 | i915_request_get(request[idx]); | 884 | i915_request_get(request[idx]); |
| 883 | i915_request_add(request[idx]); | 885 | i915_request_add(request[idx]); |
| 884 | idx++; | 886 | idx++; |
| @@ -993,13 +995,6 @@ static int live_sequential_engines(void *arg) · 108 unmodified lines | |||
| 993 | } | 995 | } |
| 994 | } | 996 | } |
| 995 | 997 | ||
| 996 | err = engine->emit_bb_start(request[idx], | ||
| 997 | batch->node.start, | ||
| 998 | batch->node.size, | ||
| 999 | 0); | ||
| 1000 | GEM_BUG_ON(err); | ||
| 1001 | request[idx]->batch = batch; | ||
| 1002 | |||
| 1003 | i915_vma_lock(batch); | 998 | i915_vma_lock(batch); |
| 1004 | err = i915_request_await_object(request[idx], | 999 | err = i915_request_await_object(request[idx], |
| 1005 | batch->obj, false); | 1000 | batch->obj, false); |
| @@ -1008,6 +1003,13 @@ static int live_sequential_engines(void *arg) · 2 unmodified lines | |||
| 1008 | i915_vma_unlock(batch); | 1003 | i915_vma_unlock(batch); |
| 1009 | GEM_BUG_ON(err); | 1004 | GEM_BUG_ON(err); |
| 1010 | 1005 | ||
| 1006 | err = engine->emit_bb_start(request[idx], | ||
| 1007 | batch->node.start, | ||
| 1008 | batch->node.size, | ||
| 1009 | 0); | ||
| 1010 | GEM_BUG_ON(err); | ||
| 1011 | request[idx]->batch = batch; | ||
| 1012 | |||
| 1011 | i915_request_get(request[idx]); | 1013 | i915_request_get(request[idx]); |
| 1012 | i915_request_add(request[idx]); | 1014 | i915_request_add(request[idx]); |
| 1013 | 1015 | ||
| @@ -1060,9 +1062,12 @@ out_request: · 46 unmodified lines | |||
| 1060 | I915_MAP_WC); | 1062 | I915_MAP_WC); |
| 1061 | if (!IS_ERR(cmd)) { | 1063 | if (!IS_ERR(cmd)) { |
| 1062 | *cmd = MI_BATCH_BUFFER_END; | 1064 | *cmd = MI_BATCH_BUFFER_END; |
| 1063 | intel_gt_chipset_flush(engine->gt); | ||
| 1064 | 1065 | ||
| 1066 | __i915_gem_object_flush_map(request[idx]->batch->obj, | ||
| 1067 | 0, sizeof(*cmd)); | ||
| 1065 | i915_gem_object_unpin_map(request[idx]->batch->obj); | 1068 | i915_gem_object_unpin_map(request[idx]->batch->obj); |
| 1069 | |||
| 1070 | intel_gt_chipset_flush(engine->gt); | ||
| 1066 | } | 1071 | } |
| 1067 | 1072 | ||
| 1068 | i915_vma_put(request[idx]->batch); | 1073 | i915_vma_put(request[idx]->batch); |
| @@ -396,6 +396,35 @@ bool __igt_timeout(unsigned long timeout, const char *fmt, ...) · 395 unmodified lines | |||
| 396 | return true; | 396 | return true; |
| 397 | } | 397 | } |
| 398 | 398 | ||
| 399 | void igt_hexdump(const void *buf, size_t len) | ||
| 400 | { | ||
| 401 | const size_t rowsize = 8 * sizeof(u32); | ||
| 402 | const void *prev = NULL; | ||
| 403 | bool skip = false; | ||
| 404 | size_t pos; | ||
| 405 | |||
| 406 | for (pos = 0; pos < len; pos += rowsize) { | ||
| 407 | char line[128]; | ||
| 408 | |||
| 409 | if (prev && !memcmp(prev, buf + pos, rowsize)) { | ||
| 410 | if (!skip) { | ||
| 411 | pr_info("*\n"); | ||
| 412 | skip = true; | ||
| 413 | } | ||
| 414 | continue; | ||
| 415 | } | ||
| 416 | |||
| 417 | WARN_ON_ONCE(hex_dump_to_buffer(buf + pos, len - pos, | ||
| 418 | rowsize, sizeof(u32), | ||
| 419 | line, sizeof(line), | ||
| 420 | false) >= sizeof(line)); | ||
| 421 | pr_info("[%04zx] %s\n", pos, line); | ||
| 422 | |||
| 423 | prev = buf + pos; | ||
| 424 | skip = false; | ||
| 425 | } | ||
| 426 | } | ||
| 427 | |||
| 399 | module_param_named(st_random_seed, i915_selftest.random_seed, uint, 0400); | 428 | module_param_named(st_random_seed, i915_selftest.random_seed, uint, 0400); |
| 400 | module_param_named(st_timeout, i915_selftest.timeout_ms, uint, 0400); | 429 | module_param_named(st_timeout, i915_selftest.timeout_ms, uint, 0400); |
| 401 | module_param_named(st_filter, i915_selftest.filter, charp, 0400); | 430 | module_param_named(st_filter, i915_selftest.filter, charp, 0400); |
| @@ -169,8 +169,7 @@ igt_spinner_create_request(struct igt_spinner *spin, · 168 unmodified lines | |||
| 169 | 169 | ||
| 170 | intel_gt_chipset_flush(engine->gt); | 170 | intel_gt_chipset_flush(engine->gt); |
| 171 | 171 | ||
| 172 | if (engine->emit_init_breadcrumb && | 172 | if (engine->emit_init_breadcrumb) { |
| 173 | i915_request_timeline(rq)->has_initial_breadcrumb) { | ||
| 174 | err = engine->emit_init_breadcrumb(rq); | 173 | err = engine->emit_init_breadcrumb(rq); |
| 175 | if (err) | 174 | if (err) |
| 176 | goto cancel_rq; | 175 | goto cancel_rq; |
| @@ -1768,20 +1768,21 @@ hdmi_vendor_any_infoframe_unpack(union hdmi_vendor_any_infoframe *frame, · 1767 unmodified lines | |||
| 1768 | } | 1768 | } |
| 1769 | 1769 | ||
| 1770 | /** | 1770 | /** |
| 1771 | * hdmi_drm_infoframe_unpack() - unpack binary buffer to a HDMI DRM infoframe | 1771 | * hdmi_drm_infoframe_unpack_only() - unpack binary buffer of CTA-861-G DRM |
| 1772 | * infoframe DataBytes to a HDMI DRM | ||
| 1773 | * infoframe | ||
| 1772 | * @frame: HDMI DRM infoframe | 1774 | * @frame: HDMI DRM infoframe |
| 1773 | * @buffer: source buffer | 1775 | * @buffer: source buffer |
| 1774 | * @size: size of buffer | 1776 | * @size: size of buffer |
| 1775 | * | 1777 | * |
| 1776 | * Unpacks the information contained in binary @buffer into a structured | 1778 | * Unpacks CTA-861-G DRM infoframe DataBytes contained in the binary @buffer |
| 1777 | * @frame of the HDMI Dynamic Range and Mastering (DRM) information frame. | 1779 | * into a structured @frame of the HDMI Dynamic Range and Mastering (DRM) |
| 1778 | * Also verifies the checksum as required by section 5.3.5 of the HDMI 1.4 | 1780 | * infoframe. |
| 1779 | * specification. | ||
| 1780 | * | 1781 | * |
| 1781 | * Returns 0 on success or a negative error code on failure. | 1782 | * Returns 0 on success or a negative error code on failure. |
| 1782 | */ | 1783 | */ |
| 1783 | static int hdmi_drm_infoframe_unpack(struct hdmi_drm_infoframe *frame, | 1784 | int hdmi_drm_infoframe_unpack_only(struct hdmi_drm_infoframe *frame, |
| 1784 | const void *buffer, size_t size) | 1785 | const void *buffer, size_t size) |
| 1785 | { | 1786 | { |
| 1786 | const u8 *ptr = buffer; | 1787 | const u8 *ptr = buffer; |
| 1787 | const u8 *temp; | 1788 | const u8 *temp; |
| @@ -1790,23 +1791,13 @@ static int hdmi_drm_infoframe_unpack(struct hdmi_drm_infoframe *frame, · 2 unmodified lines | |||
| 1790 | int ret; | 1791 | int ret; |
| 1791 | int i; | 1792 | int i; |
| 1792 | 1793 | ||
| 1793 | if (size < HDMI_INFOFRAME_SIZE(DRM)) | 1794 | if (size < HDMI_DRM_INFOFRAME_SIZE) |
| 1794 | return -EINVAL; | ||
| 1795 | |||
| 1796 | if (ptr[0] != HDMI_INFOFRAME_TYPE_DRM || | ||
| 1797 | ptr[1] != 1 || | ||
| 1798 | ptr[2] != HDMI_DRM_INFOFRAME_SIZE) | ||
| 1799 | return -EINVAL; | ||
| 1800 | |||
| 1801 | if (hdmi_infoframe_checksum(buffer, HDMI_INFOFRAME_SIZE(DRM)) != 0) | ||
| 1802 | return -EINVAL; | 1795 | return -EINVAL; |
| 1803 | 1796 | ||
| 1804 | ret = hdmi_drm_infoframe_init(frame); | 1797 | ret = hdmi_drm_infoframe_init(frame); |
| 1805 | if (ret) | 1798 | if (ret) |
| 1806 | return ret; | 1799 | return ret; |
| 1807 | 1800 | ||
| 1808 | ptr += HDMI_INFOFRAME_HEADER_SIZE; | ||
| 1809 | |||
| 1810 | frame->eotf = ptr[0] & 0x7; | 1801 | frame->eotf = ptr[0] & 0x7; |
| 1811 | frame->metadata_type = ptr[1] & 0x7; | 1802 | frame->metadata_type = ptr[1] & 0x7; |
| 1812 | 1803 | ||
| @@ -1814,7 +1805,7 @@ static int hdmi_drm_infoframe_unpack(struct hdmi_drm_infoframe *frame, · 1 unmodified line | |||
| 1814 | for (i = 0; i < 3; i++) { | 1805 | for (i = 0; i < 3; i++) { |
| 1815 | x_lsb = *temp++; | 1806 | x_lsb = *temp++; |
| 1816 | x_msb = *temp++; | 1807 | x_msb = *temp++; |
| 1817 | frame->display_primaries[i].x = (x_msb << 8) | x_lsb; | 1808 | frame->display_primaries[i].x = (x_msb << 8) | x_lsb; |
| 1818 | y_lsb = *temp++; | 1809 | y_lsb = *temp++; |
| 1819 | y_msb = *temp++; | 1810 | y_msb = *temp++; |
| 1820 | frame->display_primaries[i].y = (y_msb << 8) | y_lsb; | 1811 | frame->display_primaries[i].y = (y_msb << 8) | y_lsb; |
| @@ -1830,6 +1821,42 @@ static int hdmi_drm_infoframe_unpack(struct hdmi_drm_infoframe *frame, · 9 unmodified lines | |||
| 1830 | 1821 | ||
| 1831 | return 0; | 1822 | return 0; |
| 1832 | } | 1823 | } |
| 1824 | EXPORT_SYMBOL(hdmi_drm_infoframe_unpack_only); | ||
| 1825 | |||
| 1826 | /** | ||
| 1827 | * hdmi_drm_infoframe_unpack() - unpack binary buffer to a HDMI DRM infoframe | ||
| 1828 | * @frame: HDMI DRM infoframe | ||
| 1829 | * @buffer: source buffer | ||
| 1830 | * @size: size of buffer | ||
| 1831 | * | ||
| 1832 | * Unpacks the CTA-861-G DRM infoframe contained in the binary @buffer into | ||
| 1833 | * a structured @frame of the HDMI Dynamic Range and Mastering (DRM) | ||
| 1834 | * infoframe. It also verifies the checksum as required by section 5.3.5 of | ||
| 1835 | * the HDMI 1.4 specification. | ||
| 1836 | * | ||
| 1837 | * Returns 0 on success or a negative error code on failure. | ||
| 1838 | */ | ||
| 1839 | static int hdmi_drm_infoframe_unpack(struct hdmi_drm_infoframe *frame, | ||
| 1840 | const void *buffer, size_t size) | ||
| 1841 | { | ||
| 1842 | const u8 *ptr = buffer; | ||
| 1843 | int ret; | ||
| 1844 | |||
| 1845 | if (size < HDMI_INFOFRAME_SIZE(DRM)) | ||
| 1846 | return -EINVAL; | ||
| 1847 | |||
| 1848 | if (ptr[0] != HDMI_INFOFRAME_TYPE_DRM || | ||
| 1849 | ptr[1] != 1 || | ||
| 1850 | ptr[2] != HDMI_DRM_INFOFRAME_SIZE) | ||
| 1851 | return -EINVAL; | ||
| 1852 | |||
| 1853 | if (hdmi_infoframe_checksum(buffer, HDMI_INFOFRAME_SIZE(DRM)) != 0) | ||
| 1854 | return -EINVAL; | ||
| 1855 | |||
| 1856 | ret = hdmi_drm_infoframe_unpack_only(frame, ptr + HDMI_INFOFRAME_HEADER_SIZE, | ||
| 1857 | size - HDMI_INFOFRAME_HEADER_SIZE); | ||
| 1858 | return ret; | ||
| 1859 | } | ||
| 1833 | 1860 | ||
| 1834 | /** | 1861 | /** |
| 1835 | * hdmi_infoframe_unpack() - unpack binary buffer to a HDMI infoframe | 1862 | * hdmi_infoframe_unpack() - unpack binary buffer to a HDMI infoframe |
| @@ -1348,6 +1348,9 @@ struct drm_dp_vsc_sdp { · 1347 unmodified lines | |||
| 1348 | enum dp_content_type content_type; | 1348 | enum dp_content_type content_type; |
| 1349 | }; | 1349 | }; |
| 1350 | 1350 | ||
| 1351 | void drm_dp_vsc_sdp_log(const char *level, struct device *dev, | ||
| 1352 | const struct drm_dp_vsc_sdp *vsc); | ||
| 1353 | |||
| 1351 | int drm_dp_psr_setup_time(const u8 psr_cap[EDP_PSR_RECEIVER_CAP_SIZE]); | 1354 | int drm_dp_psr_setup_time(const u8 psr_cap[EDP_PSR_RECEIVER_CAP_SIZE]); |
| 1352 | 1355 | ||
| 1353 | static inline int | 1356 | static inline int |
| @@ -219,6 +219,8 @@ ssize_t hdmi_drm_infoframe_pack(struct hdmi_drm_infoframe *frame, void *buffer, · 218 unmodified lines | |||
| 219 | ssize_t hdmi_drm_infoframe_pack_only(const struct hdmi_drm_infoframe *frame, | 219 | ssize_t hdmi_drm_infoframe_pack_only(const struct hdmi_drm_infoframe *frame, |
| 220 | void *buffer, size_t size); | 220 | void *buffer, size_t size); |
| 221 | int hdmi_drm_infoframe_check(struct hdmi_drm_infoframe *frame); | 221 | int hdmi_drm_infoframe_check(struct hdmi_drm_infoframe *frame); |
| 222 | int hdmi_drm_infoframe_unpack_only(struct hdmi_drm_infoframe *frame, | ||
| 223 | const void *buffer, size_t size); | ||
| 222 | 224 | ||
| 223 | enum hdmi_spd_sdi { | 225 | enum hdmi_spd_sdi { |
| 224 | HDMI_SPD_SDI_UNKNOWN, | 226 | HDMI_SPD_SDI_UNKNOWN, |