History

Merge tag 'misc-habanalabs-next-2021-04-10' of https://git.kernel.org/pub/scm/linux/kernel/git/ogabbay/linux into char-misc-next

Oded writes: This tag contains habanalabs driver changes for v5.13: - Add support to reset device after the user closes the file descriptor. Because we support a single user, we can reset the device (if needs to) after a user closes its file descriptor to make sure the device is in idle and clean state for the next user. - Add a new feature to allow the user to wait on interrupt. This is needed for future ASICs - Replace GFP_ATOMIC with GFP_KERNEL wherever possible and add code to support failure of allocating with GFP_ATOMIC. - Update code to support the latest firmware image: - More security features are done in the firmware - Remove hard-coded assumptions and replace them with values that are sent to the firmware on loading. - Print device unusable error - Reset device in case the communication between driver and firmware gets out of sync. - Support new PCI device ids for secured GAUDI. - Expose current power draw through the INFO IOCTL. - Support resetting the device upon a request from the BMC (through F/W). - Always use only a single MSI in GAUDI, due to H/W limitation. - Improve data-path code by taking out code from spinlock protection. - Allow user to specify custom timeout per Command Submission. - Some enhancements to debugfs. - Various minor changes and improvements. * tag 'misc-habanalabs-next-2021-04-10' of https://git.kernel.org/pub/scm/linux/kernel/git/ogabbay/linux: (41 commits) habanalabs: print f/w boot unknown error habanalabs: update to latest F/W communication header habanalabs/gaudi: skip iATU if F/W security is enabled habanalabs/gaudi: derive security status from pci id habanalabs: move dram scrub to free sequence habanalabs: send dynamic msi-x indexes to f/w habanalabs/gaudi: clear QM errors only if not in stop_on_err mode habanalabs: support DEVICE_UNUSABLE error indication from FW habanalabs: use strscpy instead of sprintf and strlcpy habanalabs: remove the store jobs array from CS IOCTL habanalabs/gaudi: add debugfs to DMA from the device habanalabs/gaudi: sync stream add protection to SOB reset flow habanalabs: add custom timeout flag per cs habanalabs: improve utilization calculation habanalabs: support legacy and new pll indexes habanalabs: move relevant datapath work outside cs lock habanalabs: avoid soft lockup bug upon mapping error habanalabs/gaudi: Update async events header habanalabs/gaudi: unsecure TPC cfg status registers habanalabs/gaudi: always use single-msi mode ...

Greg Kroah-Hartman committed 5 years ago

Commit aa87e312 parents 19ab233 b575a7631 files · +2193 −517Browse files

Unified · Split

31 files
  1. Documentation/ABI/testing/debugfs-driver-habanalabs +54 −16
  2. drivers/misc/habanalabs/common/command_buffer.c +9 −3
  3. drivers/misc/habanalabs/common/command_submission.c +302 −66
  4. drivers/misc/habanalabs/common/context.c +11 −3
  5. drivers/misc/habanalabs/common/debugfs.c +201 −23
  6. drivers/misc/habanalabs/common/device.c +94 −127
  7. drivers/misc/habanalabs/common/firmware_if.c +216 −22
  8. drivers/misc/habanalabs/common/habanalabs.h +143 −41
  9. drivers/misc/habanalabs/common/habanalabs_drv.c +24 −4
  10. drivers/misc/habanalabs/common/habanalabs_ioctl.c +26 −9
  11. drivers/misc/habanalabs/common/hw_queue.c +1 −9
  12. drivers/misc/habanalabs/common/irq.c +56 −0
  13. drivers/misc/habanalabs/common/memory.c +129 −53
  14. drivers/misc/habanalabs/common/mmu/mmu.c +3 −0
  15. drivers/misc/habanalabs/common/pci/pci.c +52 −0
  16. drivers/misc/habanalabs/common/sysfs.c +24 −9
  17. drivers/misc/habanalabs/gaudi/gaudi.c +314 −43
  18. drivers/misc/habanalabs/gaudi/gaudiP.h +3 −0
  19. drivers/misc/habanalabs/gaudi/gaudi_security.c +0 −8
  20. drivers/misc/habanalabs/goya/goya.c +123 −17
  21. drivers/misc/habanalabs/goya/goyaP.h +2 −0
  22. drivers/misc/habanalabs/include/common/cpucp_if.h +98 −1
  23. drivers/misc/habanalabs/include/common/hl_boot_if.h +219 −0
  24. drivers/misc/habanalabs/include/gaudi/gaudi.h +1 −1
  25. drivers/misc/habanalabs/include/gaudi/gaudi_async_events.h +2 −0
  26. drivers/misc/habanalabs/include/gaudi/gaudi_async_ids_map_extended.h +27 −16
  27. drivers/misc/habanalabs/include/gaudi/gaudi_fw_if.h +0 −14
  28. drivers/misc/habanalabs/include/goya/goya.h +1 −1
  29. drivers/misc/habanalabs/include/goya/goya_async_events.h +1 −0
  30. drivers/misc/habanalabs/include/goya/goya_fw_if.h +0 −11
  31. include/uapi/misc/habanalabs.h +57 −20
Documentation/ABI/testing/debugfs-driver-habanalabs+54−16
@@ -82,6 +82,24 @@ Description: Allows the root user to read or write 64 bit data directly · 81 unmodified lines
8282 If the IOMMU is disabled, it also allows the root user to read
8383 or write from the host a device VA of a host mapped memory
8484
85+What: /sys/kernel/debug/habanalabs/hl<n>/data_dma
86+Date: Apr 2021
87+KernelVersion: 5.13
88+Contact: ogabbay@kernel.org
89+Description: Allows the root user to read from the device's internal
90+ memory (DRAM/SRAM) through a DMA engine.
91+ This property is a binary blob that contains the result of the
92+ DMA transfer.
93+ This custom interface is needed (instead of using the generic
94+ Linux user-space PCI mapping) because the amount of internal
95+ memory is huge (>32GB) and reading it via the PCI bar will take
96+ a very long time.
97+ This interface doesn't support concurrency in the same device.
98+ In GAUDI and GOYA, this action can cause undefined behavior
99+ in case the it is done while the device is executing user
100+ workloads.
101+ Only supported on GAUDI at this stage.
102+
85103 What: /sys/kernel/debug/habanalabs/hl<n>/device
86104 Date: Jan 2019
87105 KernelVersion: 5.1
@@ -90,6 +108,24 @@ Description: Enables the root user to set the device to specific state. · 2 unmodified lines
90108 Valid values are "disable", "enable", "suspend", "resume".
91109 User can read this property to see the valid values
92110
111+What: /sys/kernel/debug/habanalabs/hl<n>/dma_size
112+Date: Apr 2021
113+KernelVersion: 5.13
114+Contact: ogabbay@kernel.org
115+Description: Specify the size of the DMA transaction when using DMA to read
116+ from the device's internal memory. The value can not be larger
117+ than 128MB. Writing to this value initiates the DMA transfer.
118+ When the write is finished, the user can read the "data_dma"
119+ blob
120+
121+What: /sys/kernel/debug/habanalabs/hl<n>/dump_security_violations
122+Date: Jan 2021
123+KernelVersion: 5.12
124+Contact: ogabbay@kernel.org
125+Description: Dumps all security violations to dmesg. This will also ack
126+ all security violations meanings those violations will not be
127+ dumped next time user calls this API
128+
93129 What: /sys/kernel/debug/habanalabs/hl<n>/engines
94130 Date: Jul 2019
95131 KernelVersion: 5.3
@@ -154,6 +190,16 @@ Description: Displays the hop values and physical address for a given ASID · 58 unmodified lines
154190 e.g. to display info about VA 0x1000 for ASID 1 you need to do:
155191 echo "1 0x1000" > /sys/kernel/debug/habanalabs/hl0/mmu
156192
193+What: /sys/kernel/debug/habanalabs/hl<n>/mmu_error
194+Date: Mar 2021
195+KernelVersion: 5.12
196+Contact: fkassabri@habana.ai
197+Description: Check and display page fault or access violation mmu errors for
198+ all MMUs specified in mmu_cap_mask.
199+ e.g. to display error info for MMU hw cap bit 9, you need to do:
200+ echo "0x200" > /sys/kernel/debug/habanalabs/hl0/mmu_error
201+ cat /sys/kernel/debug/habanalabs/hl0/mmu_error
202+
157203 What: /sys/kernel/debug/habanalabs/hl<n>/set_power_state
158204 Date: Jan 2019
159205 KernelVersion: 5.1
@@ -161,6 +207,13 @@ Contact: ogabbay@kernel.org · 1 unmodified line
161207 Description: Sets the PCI power state. Valid values are "1" for D0 and "2"
162208 for D3Hot
163209
210+What: /sys/kernel/debug/habanalabs/hl<n>/stop_on_err
211+Date: Mar 2020
212+KernelVersion: 5.6
213+Contact: ogabbay@kernel.org
214+Description: Sets the stop-on_error option for the device engines. Value of
215+ "0" is for disable, otherwise enable.
216+
164217 What: /sys/kernel/debug/habanalabs/hl<n>/userptr
165218 Date: Jan 2019
166219 KernelVersion: 5.1
@@ -174,19 +227,4 @@ Date: Jan 2019 · 7 unmodified lines
174227 KernelVersion: 5.1
175228 Contact: ogabbay@kernel.org
176229 Description: Displays a list with information about all the active virtual
177− address mappings per ASID
178−
179−What: /sys/kernel/debug/habanalabs/hl<n>/stop_on_err
180−Date: Mar 2020
181−KernelVersion: 5.6
182−Contact: ogabbay@kernel.org
183−Description: Sets the stop-on_error option for the device engines. Value of
184− "0" is for disable, otherwise enable.
185−
186−What: /sys/kernel/debug/habanalabs/hl<n>/dump_security_violations
187−Date: Jan 2021
188−KernelVersion: 5.12
189−Contact: ogabbay@kernel.org
190−Description: Dumps all security violations to dmesg. This will also ack
191− all security violations meanings those violations will not be
192− dumped next time user calls this API
230+ address mappings per ASID and all user mappings of HW blocks
drivers/misc/habanalabs/common/command_buffer.c+9−3
@@ -181,7 +181,7 @@ static void cb_release(struct kref *ref) · 180 unmodified lines
181181 static struct hl_cb *hl_cb_alloc(struct hl_device *hdev, u32 cb_size,
182182 int ctx_id, bool internal_cb)
183183 {
184− struct hl_cb *cb;
184+ struct hl_cb *cb = NULL;
185185 u32 cb_offset;
186186 void *p;
187187
@@ -193,9 +193,10 @@ static struct hl_cb *hl_cb_alloc(struct hl_device *hdev, u32 cb_size, · 5 unmodified lines
193193 * the kernel's copy. Hence, we must never sleep in this code section
194194 * and must use GFP_ATOMIC for all memory allocations.
195195 */
196− if (ctx_id == HL_KERNEL_ASID_ID)
196+ if (ctx_id == HL_KERNEL_ASID_ID && !hdev->disabled)
197197 cb = kzalloc(sizeof(*cb), GFP_ATOMIC);
198− else
198+
199+ if (!cb)
199200 cb = kzalloc(sizeof(*cb), GFP_KERNEL);
200201
201202 if (!cb)
@@ -214,6 +215,9 @@ static struct hl_cb *hl_cb_alloc(struct hl_device *hdev, u32 cb_size, · 12 unmodified lines
214215 } else if (ctx_id == HL_KERNEL_ASID_ID) {
215216 p = hdev->asic_funcs->asic_dma_alloc_coherent(hdev, cb_size,
216217 &cb->bus_address, GFP_ATOMIC);
218+ if (!p)
219+ p = hdev->asic_funcs->asic_dma_alloc_coherent(hdev,
220+ cb_size, &cb->bus_address, GFP_KERNEL);
217221 } else {
218222 p = hdev->asic_funcs->asic_dma_alloc_coherent(hdev, cb_size,
219223 &cb->bus_address,
@@ -310,6 +314,8 @@ int hl_cb_create(struct hl_device *hdev, struct hl_cb_mgr *mgr, · 90 unmodified lines
310314
311315 spin_lock(&mgr->cb_lock);
312316 rc = idr_alloc(&mgr->cb_handles, cb, 1, 0, GFP_ATOMIC);
317+ if (rc < 0)
318+ rc = idr_alloc(&mgr->cb_handles, cb, 1, 0, GFP_KERNEL);
313319 spin_unlock(&mgr->cb_lock);
314320
315321 if (rc < 0) {
drivers/misc/habanalabs/common/command_submission.c+302−66
@@ -84,6 +84,38 @@ int hl_gen_sob_mask(u16 sob_base, u8 sob_mask, u8 *mask) · 83 unmodified lines
8484 return 0;
8585 }
8686
87+static void sob_reset_work(struct work_struct *work)
88+{
89+ struct hl_cs_compl *hl_cs_cmpl =
90+ container_of(work, struct hl_cs_compl, sob_reset_work);
91+ struct hl_device *hdev = hl_cs_cmpl->hdev;
92+
93+ /*
94+ * A signal CS can get completion while the corresponding wait
95+ * for signal CS is on its way to the PQ. The wait for signal CS
96+ * will get stuck if the signal CS incremented the SOB to its
97+ * max value and there are no pending (submitted) waits on this
98+ * SOB.
99+ * We do the following to void this situation:
100+ * 1. The wait for signal CS must get a ref for the signal CS as
101+ * soon as possible in cs_ioctl_signal_wait() and put it
102+ * before being submitted to the PQ but after it incremented
103+ * the SOB refcnt in init_signal_wait_cs().
104+ * 2. Signal/Wait for signal CS will decrement the SOB refcnt
105+ * here.
106+ * These two measures guarantee that the wait for signal CS will
107+ * reset the SOB upon completion rather than the signal CS and
108+ * hence the above scenario is avoided.
109+ */
110+ kref_put(&hl_cs_cmpl->hw_sob->kref, hl_sob_reset);
111+
112+ if (hl_cs_cmpl->type == CS_TYPE_COLLECTIVE_WAIT)
113+ hdev->asic_funcs->reset_sob_group(hdev,
114+ hl_cs_cmpl->sob_group);
115+
116+ kfree(hl_cs_cmpl);
117+}
118+
87119 static void hl_fence_release(struct kref *kref)
88120 {
89121 struct hl_fence *fence =
@@ -109,28 +141,9 @@ static void hl_fence_release(struct kref *kref) · 19 unmodified lines
109141 hl_cs_cmpl->hw_sob->sob_id,
110142 hl_cs_cmpl->sob_val);
111143
112− /*
113− * A signal CS can get completion while the corresponding wait
114− * for signal CS is on its way to the PQ. The wait for signal CS
115− * will get stuck if the signal CS incremented the SOB to its
116− * max value and there are no pending (submitted) waits on this
117− * SOB.
118− * We do the following to void this situation:
119− * 1. The wait for signal CS must get a ref for the signal CS as
120− * soon as possible in cs_ioctl_signal_wait() and put it
121− * before being submitted to the PQ but after it incremented
122− * the SOB refcnt in init_signal_wait_cs().
123− * 2. Signal/Wait for signal CS will decrement the SOB refcnt
124− * here.
125− * These two measures guarantee that the wait for signal CS will
126− * reset the SOB upon completion rather than the signal CS and
127− * hence the above scenario is avoided.
128− */
129− kref_put(&hl_cs_cmpl->hw_sob->kref, hl_sob_reset);
144+ queue_work(hdev->sob_reset_wq, &hl_cs_cmpl->sob_reset_work);
130145
131− if (hl_cs_cmpl->type == CS_TYPE_COLLECTIVE_WAIT)
132− hdev->asic_funcs->reset_sob_group(hdev,
133− hl_cs_cmpl->sob_group);
146+ return;
134147 }
135148
136149 free:
@@ -454,8 +467,7 @@ static void cs_handle_tdr(struct hl_device *hdev, struct hl_cs *cs) · 317 unmodified lines
454467
455468 if (next_entry_found && !next->tdr_active) {
456469 next->tdr_active = true;
457− schedule_delayed_work(&next->work_tdr,
458− hdev->timeout_jiffies);
470+ schedule_delayed_work(&next->work_tdr, next->timeout_jiffies);
459471 }
460472
461473 spin_unlock(&hdev->cs_mirror_lock);
@@ -492,24 +504,6 @@ static void cs_do_release(struct kref *ref) · 30 unmodified lines
492504 goto out;
493505 }
494506
495− hdev->asic_funcs->hw_queues_lock(hdev);
496−
497− hdev->cs_active_cnt--;
498− if (!hdev->cs_active_cnt) {
499− struct hl_device_idle_busy_ts *ts;
500−
501− ts = &hdev->idle_busy_ts_arr[hdev->idle_busy_ts_idx++];
502− ts->busy_to_idle_ts = ktime_get();
503−
504− if (hdev->idle_busy_ts_idx == HL_IDLE_BUSY_TS_ARR_SIZE)
505− hdev->idle_busy_ts_idx = 0;
506− } else if (hdev->cs_active_cnt < 0) {
507− dev_crit(hdev->dev, "CS active cnt %d is negative\n",
508− hdev->cs_active_cnt);
509− }
510−
511− hdev->asic_funcs->hw_queues_unlock(hdev);
512−
513507 /* Need to update CI for all queue jobs that does not get completion */
514508 hl_hw_queue_update_ci(cs);
515509
@@ -620,14 +614,14 @@ static void cs_timedout(struct work_struct *work) · 104 unmodified lines
620614 cs_put(cs);
621615
622616 if (hdev->reset_on_lockup)
623− hl_device_reset(hdev, false, false);
617+ hl_device_reset(hdev, 0);
624618 else
625619 hdev->needs_reset = true;
626620 }
627621
628622 static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx,
629623 enum hl_cs_type cs_type, u64 user_sequence,
630− struct hl_cs **cs_new)
624+ struct hl_cs **cs_new, u32 flags, u32 timeout)
631625 {
632626 struct hl_cs_counters_atomic *cntr;
633627 struct hl_fence *other = NULL;
@@ -638,6 +632,9 @@ static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx, · 4 unmodified lines
638632 cntr = &hdev->aggregated_cs_counters;
639633
640634 cs = kzalloc(sizeof(*cs), GFP_ATOMIC);
635+ if (!cs)
636+ cs = kzalloc(sizeof(*cs), GFP_KERNEL);
637+
641638 if (!cs) {
642639 atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
643640 atomic64_inc(&cntr->out_of_mem_drop_cnt);
@@ -651,12 +648,17 @@ static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx, · 7 unmodified lines
651648 cs->submitted = false;
652649 cs->completed = false;
653650 cs->type = cs_type;
651+ cs->timestamp = !!(flags & HL_CS_FLAGS_TIMESTAMP);
652+ cs->timeout_jiffies = timeout;
654653 INIT_LIST_HEAD(&cs->job_list);
655654 INIT_DELAYED_WORK(&cs->work_tdr, cs_timedout);
656655 kref_init(&cs->refcount);
657656 spin_lock_init(&cs->job_lock);
658657
659658 cs_cmpl = kmalloc(sizeof(*cs_cmpl), GFP_ATOMIC);
659+ if (!cs_cmpl)
660+ cs_cmpl = kmalloc(sizeof(*cs_cmpl), GFP_KERNEL);
661+
660662 if (!cs_cmpl) {
661663 atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
662664 atomic64_inc(&cntr->out_of_mem_drop_cnt);
@@ -664,9 +666,23 @@ static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx, · 1 unmodified line
664666 goto free_cs;
665667 }
666668
669+ cs->jobs_in_queue_cnt = kcalloc(hdev->asic_prop.max_queues,
670+ sizeof(*cs->jobs_in_queue_cnt), GFP_ATOMIC);
671+ if (!cs->jobs_in_queue_cnt)
672+ cs->jobs_in_queue_cnt = kcalloc(hdev->asic_prop.max_queues,
673+ sizeof(*cs->jobs_in_queue_cnt), GFP_KERNEL);
674+
675+ if (!cs->jobs_in_queue_cnt) {
676+ atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
677+ atomic64_inc(&cntr->out_of_mem_drop_cnt);
678+ rc = -ENOMEM;
679+ goto free_cs_cmpl;
680+ }
681+
667682 cs_cmpl->hdev = hdev;
668683 cs_cmpl->type = cs->type;
669684 spin_lock_init(&cs_cmpl->lock);
685+ INIT_WORK(&cs_cmpl->sob_reset_work, sob_reset_work);
670686 cs->fence = &cs_cmpl->base_fence;
671687
672688 spin_lock(&ctx->cs_lock);
@@ -696,15 +712,6 @@ static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx, · 23 unmodified lines
696712 goto free_fence;
697713 }
698714
699− cs->jobs_in_queue_cnt = kcalloc(hdev->asic_prop.max_queues,
700− sizeof(*cs->jobs_in_queue_cnt), GFP_ATOMIC);
701− if (!cs->jobs_in_queue_cnt) {
702− atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
703− atomic64_inc(&cntr->out_of_mem_drop_cnt);
704− rc = -ENOMEM;
705− goto free_fence;
706− }
707−
708715 /* init hl_fence */
709716 hl_fence_init(&cs_cmpl->base_fence, cs_cmpl->cs_seq);
710717
@@ -727,6 +734,8 @@ static int allocate_cs(struct hl_device *hdev, struct hl_ctx *ctx, · 16 unmodified lines
727734
728735 free_fence:
729736 spin_unlock(&ctx->cs_lock);
737+ kfree(cs->jobs_in_queue_cnt);
738+free_cs_cmpl:
730739 kfree(cs_cmpl);
731740 free_cs:
732741 kfree(cs);
@@ -749,6 +758,8 @@ void hl_cs_rollback_all(struct hl_device *hdev) · 16 unmodified lines
749758 int i;
750759 struct hl_cs *cs, *tmp;
751760
761+ flush_workqueue(hdev->sob_reset_wq);
762+
752763 /* flush all completions before iterating over the CS mirror list in
753764 * order to avoid a race with the release functions
754765 */
@@ -778,6 +789,44 @@ void hl_pending_cb_list_flush(struct hl_ctx *ctx) · 23 unmodified lines
778789 }
779790 }
780791
792+static void
793+wake_pending_user_interrupt_threads(struct hl_user_interrupt *interrupt)
794+{
795+ struct hl_user_pending_interrupt *pend;
796+
797+ spin_lock(&interrupt->wait_list_lock);
798+ list_for_each_entry(pend, &interrupt->wait_list_head, wait_list_node) {
799+ pend->fence.error = -EIO;
800+ complete_all(&pend->fence.completion);
801+ }
802+ spin_unlock(&interrupt->wait_list_lock);
803+}
804+
805+void hl_release_pending_user_interrupts(struct hl_device *hdev)
806+{
807+ struct asic_fixed_properties *prop = &hdev->asic_prop;
808+ struct hl_user_interrupt *interrupt;
809+ int i;
810+
811+ if (!prop->user_interrupt_count)
812+ return;
813+
814+ /* We iterate through the user interrupt requests and waking up all
815+ * user threads waiting for interrupt completion. We iterate the
816+ * list under a lock, this is why all user threads, once awake,
817+ * will wait on the same lock and will release the waiting object upon
818+ * unlock.
819+ */
820+
821+ for (i = 0 ; i < prop->user_interrupt_count ; i++) {
822+ interrupt = &hdev->user_interrupt[i];
823+ wake_pending_user_interrupt_threads(interrupt);
824+ }
825+
826+ interrupt = &hdev->common_user_interrupt;
827+ wake_pending_user_interrupt_threads(interrupt);
828+}
829+
781830 static void job_wq_completion(struct work_struct *work)
782831 {
783832 struct hl_cs_job *job = container_of(work, struct hl_cs_job,
@@ -889,6 +938,9 @@ struct hl_cs_job *hl_cs_allocate_job(struct hl_device *hdev, · 105 unmodified lines
889938 struct hl_cs_job *job;
890939
891940 job = kzalloc(sizeof(*job), GFP_ATOMIC);
941+ if (!job)
942+ job = kzalloc(sizeof(*job), GFP_KERNEL);
943+
892944 if (!job)
893945 return NULL;
894946
@@ -991,6 +1043,9 @@ static int hl_cs_copy_chunk_array(struct hl_device *hdev, · 96 unmodified lines
9911043
9921044 *cs_chunk_array = kmalloc_array(num_chunks, sizeof(**cs_chunk_array),
9931045 GFP_ATOMIC);
1046+ if (!*cs_chunk_array)
1047+ *cs_chunk_array = kmalloc_array(num_chunks,
1048+ sizeof(**cs_chunk_array), GFP_KERNEL);
9941049 if (!*cs_chunk_array) {
9951050 atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
9961051 atomic64_inc(&hdev->aggregated_cs_counters.out_of_mem_drop_cnt);
@@ -1038,7 +1093,8 @@ static int cs_staged_submission(struct hl_device *hdev, struct hl_cs *cs, · 41 unmodified lines
10381093 }
10391094
10401095 static int cs_ioctl_default(struct hl_fpriv *hpriv, void __user *chunks,
1041− u32 num_chunks, u64 *cs_seq, u32 flags)
1096+ u32 num_chunks, u64 *cs_seq, u32 flags,
1097+ u32 timeout)
10421098 {
10431099 bool staged_mid, int_queues_only = true;
10441100 struct hl_device *hdev = hpriv->hdev;
@@ -1067,11 +1123,11 @@ static int cs_ioctl_default(struct hl_fpriv *hpriv, void __user *chunks, · 22 unmodified lines
10671123 staged_mid = false;
10681124
10691125 rc = allocate_cs(hdev, hpriv->ctx, CS_TYPE_DEFAULT,
1070− staged_mid ? user_sequence : ULLONG_MAX, &cs);
1126+ staged_mid ? user_sequence : ULLONG_MAX, &cs, flags,
1127+ timeout);
10711128 if (rc)
10721129 goto free_cs_chunk_array;
10731130
1074− cs->timestamp = !!(flags & HL_CS_FLAGS_TIMESTAMP);
10751131 *cs_seq = cs->sequence;
10761132
10771133 hl_debugfs_add_cs(cs);
@@ -1269,7 +1325,8 @@ static int hl_submit_pending_cb(struct hl_fpriv *hpriv) · 191 unmodified lines
12691325 list_move_tail(&pending_cb->cb_node, &local_cb_list);
12701326 spin_unlock(&ctx->pending_cb_lock);
12711327
1272− rc = allocate_cs(hdev, ctx, CS_TYPE_DEFAULT, ULLONG_MAX, &cs);
1328+ rc = allocate_cs(hdev, ctx, CS_TYPE_DEFAULT, ULLONG_MAX, &cs, 0,
1329+ hdev->timeout_jiffies);
12731330 if (rc)
12741331 goto add_list_elements;
12751332
@@ -1370,7 +1427,7 @@ static int hl_cs_ctx_switch(struct hl_fpriv *hpriv, union hl_cs_args *args, · 94 unmodified lines
13701427 rc = 0;
13711428 } else {
13721429 rc = cs_ioctl_default(hpriv, chunks, num_chunks,
1373− cs_seq, 0);
1430+ cs_seq, 0, hdev->timeout_jiffies);
13741431 }
13751432
13761433 mutex_unlock(&hpriv->restore_phase_mutex);
@@ -1419,7 +1476,7 @@ wait_again: · 42 unmodified lines
14191476
14201477 out:
14211478 if ((rc == -ETIMEDOUT || rc == -EBUSY) && (need_soft_reset))
1422− hl_device_reset(hdev, false, false);
1479+ hl_device_reset(hdev, 0);
14231480
14241481 return rc;
14251482 }
@@ -1445,6 +1502,10 @@ static int cs_ioctl_extract_signal_seq(struct hl_device *hdev, · 19 unmodified lines
14451502 signal_seq_arr = kmalloc_array(signal_seq_arr_len,
14461503 sizeof(*signal_seq_arr),
14471504 GFP_ATOMIC);
1505+ if (!signal_seq_arr)
1506+ signal_seq_arr = kmalloc_array(signal_seq_arr_len,
1507+ sizeof(*signal_seq_arr),
1508+ GFP_KERNEL);
14481509 if (!signal_seq_arr) {
14491510 atomic64_inc(&ctx->cs_counters.out_of_mem_drop_cnt);
14501511 atomic64_inc(&hdev->aggregated_cs_counters.out_of_mem_drop_cnt);
@@ -1536,7 +1597,7 @@ static int cs_ioctl_signal_wait_create_jobs(struct hl_device *hdev, · 85 unmodified lines
15361597
15371598 static int cs_ioctl_signal_wait(struct hl_fpriv *hpriv, enum hl_cs_type cs_type,
15381599 void __user *chunks, u32 num_chunks,
1539− u64 *cs_seq, bool timestamp)
1600+ u64 *cs_seq, u32 flags, u32 timeout)
15401601 {
15411602 struct hl_cs_chunk *cs_chunk_array, *chunk;
15421603 struct hw_queue_properties *hw_queue_prop;
@@ -1642,7 +1703,7 @@ static int cs_ioctl_signal_wait(struct hl_fpriv *hpriv, enum hl_cs_type cs_type, · 99 unmodified lines
16421703 }
16431704 }
16441705
1645− rc = allocate_cs(hdev, ctx, cs_type, ULLONG_MAX, &cs);
1706+ rc = allocate_cs(hdev, ctx, cs_type, ULLONG_MAX, &cs, flags, timeout);
16461707 if (rc) {
16471708 if (cs_type == CS_TYPE_WAIT ||
16481709 cs_type == CS_TYPE_COLLECTIVE_WAIT)
@@ -1650,8 +1711,6 @@ static int cs_ioctl_signal_wait(struct hl_fpriv *hpriv, enum hl_cs_type cs_type, · 1 unmodified line
16501711 goto free_cs_chunk_array;
16511712 }
16521713
1653− cs->timestamp = !!timestamp;
1654−
16551714 /*
16561715 * Save the signal CS fence for later initialization right before
16571716 * hanging the wait CS on the queue.
@@ -1709,7 +1768,7 @@ int hl_cs_ioctl(struct hl_fpriv *hpriv, void *data) · 51 unmodified lines
17091768 enum hl_cs_type cs_type;
17101769 u64 cs_seq = ULONG_MAX;
17111770 void __user *chunks;
1712− u32 num_chunks, flags;
1771+ u32 num_chunks, flags, timeout;
17131772 int rc;
17141773
17151774 rc = hl_cs_sanity_checks(hpriv, args);
@@ -1735,16 +1794,20 @@ int hl_cs_ioctl(struct hl_fpriv *hpriv, void *data) · 19 unmodified lines
17351794 !(flags & HL_CS_FLAGS_STAGED_SUBMISSION_FIRST))
17361795 cs_seq = args->in.seq;
17371796
1797+ timeout = flags & HL_CS_FLAGS_CUSTOM_TIMEOUT
1798+ ? msecs_to_jiffies(args->in.timeout * 1000)
1799+ : hpriv->hdev->timeout_jiffies;
1800+
17381801 switch (cs_type) {
17391802 case CS_TYPE_SIGNAL:
17401803 case CS_TYPE_WAIT:
17411804 case CS_TYPE_COLLECTIVE_WAIT:
17421805 rc = cs_ioctl_signal_wait(hpriv, cs_type, chunks, num_chunks,
1743− &cs_seq, args->in.cs_flags & HL_CS_FLAGS_TIMESTAMP);
1806+ &cs_seq, args->in.cs_flags, timeout);
17441807 break;
17451808 default:
17461809 rc = cs_ioctl_default(hpriv, chunks, num_chunks, &cs_seq,
1747− args->in.cs_flags);
1810+ args->in.cs_flags, timeout);
17481811 break;
17491812 }
17501813
@@ -1818,7 +1881,7 @@ static int _hl_cs_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, · 67 unmodified lines
18181881 return rc;
18191882 }
18201883
1821−int hl_cs_wait_ioctl(struct hl_fpriv *hpriv, void *data)
1884+static int hl_cs_wait_ioctl(struct hl_fpriv *hpriv, void *data)
18221885 {
18231886 struct hl_device *hdev = hpriv->hdev;
18241887 union hl_wait_cs_args *args = data;
@@ -1873,3 +1936,176 @@ int hl_cs_wait_ioctl(struct hl_fpriv *hpriv, void *data) · 48 unmodified lines
18731936
18741937 return 0;
18751938 }
1939+
1940+static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx,
1941+ u32 timeout_us, u64 user_address,
1942+ u32 target_value, u16 interrupt_offset,
1943+ enum hl_cs_wait_status *status)
1944+{
1945+ struct hl_user_pending_interrupt *pend;
1946+ struct hl_user_interrupt *interrupt;
1947+ unsigned long timeout;
1948+ long completion_rc;
1949+ u32 completion_value;
1950+ int rc = 0;
1951+
1952+ if (timeout_us == U32_MAX)
1953+ timeout = timeout_us;
1954+ else
1955+ timeout = usecs_to_jiffies(timeout_us);
1956+
1957+ hl_ctx_get(hdev, ctx);
1958+
1959+ pend = kmalloc(sizeof(*pend), GFP_KERNEL);
1960+ if (!pend) {
1961+ hl_ctx_put(ctx);
1962+ return -ENOMEM;
1963+ }
1964+
1965+ hl_fence_init(&pend->fence, ULONG_MAX);
1966+
1967+ if (interrupt_offset == HL_COMMON_USER_INTERRUPT_ID)
1968+ interrupt = &hdev->common_user_interrupt;
1969+ else
1970+ interrupt = &hdev->user_interrupt[interrupt_offset];
1971+
1972+ spin_lock(&interrupt->wait_list_lock);
1973+ if (!hl_device_operational(hdev, NULL)) {
1974+ rc = -EPERM;
1975+ goto unlock_and_free_fence;
1976+ }
1977+
1978+ if (copy_from_user(&completion_value, u64_to_user_ptr(user_address), 4)) {
1979+ dev_err(hdev->dev,
1980+ "Failed to copy completion value from user\n");
1981+ rc = -EFAULT;
1982+ goto unlock_and_free_fence;
1983+ }
1984+
1985+ if (completion_value >= target_value)
1986+ *status = CS_WAIT_STATUS_COMPLETED;
1987+ else
1988+ *status = CS_WAIT_STATUS_BUSY;
1989+
1990+ if (!timeout_us || (*status == CS_WAIT_STATUS_COMPLETED))
1991+ goto unlock_and_free_fence;
1992+
1993+ /* Add pending user interrupt to relevant list for the interrupt
1994+ * handler to monitor
1995+ */
1996+ list_add_tail(&pend->wait_list_node, &interrupt->wait_list_head);
1997+ spin_unlock(&interrupt->wait_list_lock);
1998+
1999+wait_again:
2000+ /* Wait for interrupt handler to signal completion */
2001+ completion_rc =
2002+ wait_for_completion_interruptible_timeout(
2003+ &pend->fence.completion, timeout);
2004+
2005+ /* If timeout did not expire we need to perform the comparison.
2006+ * If comparison fails, keep waiting until timeout expires
2007+ */
2008+ if (completion_rc > 0) {
2009+ if (copy_from_user(&completion_value,
2010+ u64_to_user_ptr(user_address), 4)) {
2011+ dev_err(hdev->dev,
2012+ "Failed to copy completion value from user\n");
2013+ rc = -EFAULT;
2014+ goto remove_pending_user_interrupt;
2015+ }
2016+
2017+ if (completion_value >= target_value) {
2018+ *status = CS_WAIT_STATUS_COMPLETED;
2019+ } else {
2020+ timeout -= jiffies_to_usecs(completion_rc);
2021+ goto wait_again;
2022+ }
2023+ } else {
2024+ *status = CS_WAIT_STATUS_BUSY;
2025+ }
2026+
2027+remove_pending_user_interrupt:
2028+ spin_lock(&interrupt->wait_list_lock);
2029+ list_del(&pend->wait_list_node);
2030+
2031+unlock_and_free_fence:
2032+ spin_unlock(&interrupt->wait_list_lock);
2033+ kfree(pend);
2034+ hl_ctx_put(ctx);
2035+
2036+ return rc;
2037+}
2038+
2039+static int hl_interrupt_wait_ioctl(struct hl_fpriv *hpriv, void *data)
2040+{
2041+ u16 interrupt_id, interrupt_offset, first_interrupt, last_interrupt;
2042+ struct hl_device *hdev = hpriv->hdev;
2043+ struct asic_fixed_properties *prop;
2044+ union hl_wait_cs_args *args = data;
2045+ enum hl_cs_wait_status status;
2046+ int rc;
2047+
2048+ prop = &hdev->asic_prop;
2049+
2050+ if (!prop->user_interrupt_count) {
2051+ dev_err(hdev->dev, "no user interrupts allowed");
2052+ return -EPERM;
2053+ }
2054+
2055+ interrupt_id =
2056+ FIELD_GET(HL_WAIT_CS_FLAGS_INTERRUPT_MASK, args->in.flags);
2057+
2058+ first_interrupt = prop->first_available_user_msix_interrupt;
2059+ last_interrupt = prop->first_available_user_msix_interrupt +
2060+ prop->user_interrupt_count - 1;
2061+
2062+ if ((interrupt_id < first_interrupt || interrupt_id > last_interrupt) &&
2063+ interrupt_id != HL_COMMON_USER_INTERRUPT_ID) {
2064+ dev_err(hdev->dev, "invalid user interrupt %u", interrupt_id);
2065+ return -EINVAL;
2066+ }
2067+
2068+ if (interrupt_id == HL_COMMON_USER_INTERRUPT_ID)
2069+ interrupt_offset = HL_COMMON_USER_INTERRUPT_ID;
2070+ else
2071+ interrupt_offset = interrupt_id - first_interrupt;
2072+
2073+ rc = _hl_interrupt_wait_ioctl(hdev, hpriv->ctx,
2074+ args->in.interrupt_timeout_us, args->in.addr,
2075+ args->in.target, interrupt_offset, &status);
2076+
2077+ memset(args, 0, sizeof(*args));
2078+
2079+ if (rc) {
2080+ dev_err_ratelimited(hdev->dev,
2081+ "interrupt_wait_ioctl failed (%d)\n", rc);
2082+
2083+ return rc;
2084+ }
2085+
2086+ switch (status) {
2087+ case CS_WAIT_STATUS_COMPLETED:
2088+ args->out.status = HL_WAIT_CS_STATUS_COMPLETED;
2089+ break;
2090+ case CS_WAIT_STATUS_BUSY:
2091+ default:
2092+ args->out.status = HL_WAIT_CS_STATUS_BUSY;
2093+ break;
2094+ }
2095+
2096+ return 0;
2097+}
2098+
2099+int hl_wait_ioctl(struct hl_fpriv *hpriv, void *data)
2100+{
2101+ union hl_wait_cs_args *args = data;
2102+ u32 flags = args->in.flags;
2103+ int rc;
2104+
2105+ if (flags & HL_WAIT_CS_FLAGS_INTERRUPT)
2106+ rc = hl_interrupt_wait_ioctl(hpriv, data);
2107+ else
2108+ rc = hl_cs_wait_ioctl(hpriv, data);
2109+
2110+ return rc;
2111+}
drivers/misc/habanalabs/common/context.c+11−3
@@ -20,6 +20,11 @@ static void hl_ctx_fini(struct hl_ctx *ctx) · 19 unmodified lines
2020 */
2121 hl_pending_cb_list_flush(ctx);
2222
23+ /* Release all allocated HW block mapped list entries and destroy
24+ * the mutex.
25+ */
26+ hl_hw_block_mem_fini(ctx);
27+
2328 /*
2429 * If we arrived here, there are no jobs waiting for this context
2530 * on its queues so we can safely remove it.
@@ -160,13 +165,15 @@ int hl_ctx_init(struct hl_device *hdev, struct hl_ctx *ctx, bool is_kernel_ctx) · 134 unmodified lines
160165 if (!ctx->cs_pending)
161166 return -ENOMEM;
162167
168+ hl_hw_block_mem_init(ctx);
169+
163170 if (is_kernel_ctx) {
164171 ctx->asid = HL_KERNEL_ASID_ID; /* Kernel driver gets ASID 0 */
165172 rc = hl_vm_ctx_init(ctx);
166173 if (rc) {
167174 dev_err(hdev->dev, "Failed to init mem ctx module\n");
168175 rc = -ENOMEM;
169− goto err_free_cs_pending;
176+ goto err_hw_block_mem_fini;
170177 }
171178
172179 rc = hdev->asic_funcs->ctx_init(ctx);
@@ -179,7 +186,7 @@ int hl_ctx_init(struct hl_device *hdev, struct hl_ctx *ctx, bool is_kernel_ctx) · 6 unmodified lines
179186 if (!ctx->asid) {
180187 dev_err(hdev->dev, "No free ASID, failed to create context\n");
181188 rc = -ENOMEM;
182− goto err_free_cs_pending;
189+ goto err_hw_block_mem_fini;
183190 }
184191
185192 rc = hl_vm_ctx_init(ctx);
@@ -214,7 +221,8 @@ err_vm_ctx_fini: · 28 unmodified lines
214221 err_asid_free:
215222 if (ctx->asid != HL_KERNEL_ASID_ID)
216223 hl_asid_free(hdev, ctx->asid);
217−err_free_cs_pending:
224+err_hw_block_mem_fini:
225+ hl_hw_block_mem_fini(ctx);
218226 kfree(ctx->cs_pending);
219227
220228 return rc;
drivers/misc/habanalabs/common/debugfs.c+201−23
@@ -9,8 +9,8 @@ · 8 unmodified lines
99 #include "../include/hw_ip/mmu/mmu_general.h"
1010
1111 #include <linux/pci.h>
12−#include <linux/debugfs.h>
1312 #include <linux/uaccess.h>
13+#include <linux/vmalloc.h>
1414
1515 #define MMU_ADDR_BUF_SIZE 40
1616 #define MMU_ASID_BUF_SIZE 10
@@ -229,6 +229,7 @@ static int vm_show(struct seq_file *s, void *data) · 212 unmodified lines
229229 {
230230 struct hl_debugfs_entry *entry = s->private;
231231 struct hl_dbg_device_entry *dev_entry = entry->dev_entry;
232+ struct hl_vm_hw_block_list_node *lnode;
232233 struct hl_ctx *ctx;
233234 struct hl_vm *vm;
234235 struct hl_vm_hash_node *hnode;
@@ -272,6 +273,21 @@ static int vm_show(struct seq_file *s, void *data) · 37 unmodified lines
272273 }
273274 mutex_unlock(&ctx->mem_hash_lock);
274275
276+ if (ctx->asid != HL_KERNEL_ASID_ID &&
277+ !list_empty(&ctx->hw_block_mem_list)) {
278+ seq_puts(s, "\nhw_block mappings:\n\n");
279+ seq_puts(s, " virtual address size HW block id\n");
280+ seq_puts(s, "-------------------------------------------\n");
281+ mutex_lock(&ctx->hw_block_list_lock);
282+ list_for_each_entry(lnode, &ctx->hw_block_mem_list,
283+ node) {
284+ seq_printf(s,
285+ " 0x%-14lx %-6u %-9u\n",
286+ lnode->vaddr, lnode->size, lnode->id);
287+ }
288+ mutex_unlock(&ctx->hw_block_list_lock);
289+ }
290+
275291 vm = &ctx->hdev->vm;
276292 spin_lock(&vm->idr_lock);
277293
@@ -441,21 +457,86 @@ out: · 163 unmodified lines
441457 return false;
442458 }
443459
444−static int device_va_to_pa(struct hl_device *hdev, u64 virt_addr,
445− u64 *phys_addr)
460+static bool hl_is_device_internal_memory_va(struct hl_device *hdev, u64 addr,
461+ u32 size)
446462 {
463+ struct asic_fixed_properties *prop = &hdev->asic_prop;
464+ u64 dram_start_addr, dram_end_addr;
465+
466+ if (!hdev->mmu_enable)
467+ return false;
468+
469+ if (prop->dram_supports_virtual_memory) {
470+ dram_start_addr = prop->dmmu.start_addr;
471+ dram_end_addr = prop->dmmu.end_addr;
472+ } else {
473+ dram_start_addr = prop->dram_base_address;
474+ dram_end_addr = prop->dram_end_address;
475+ }
476+
477+ if (hl_mem_area_inside_range(addr, size, dram_start_addr,
478+ dram_end_addr))
479+ return true;
480+
481+ if (hl_mem_area_inside_range(addr, size, prop->sram_base_address,
482+ prop->sram_end_address))
483+ return true;
484+
485+ return false;
486+}
487+
488+static int device_va_to_pa(struct hl_device *hdev, u64 virt_addr, u32 size,
489+ u64 *phys_addr)
490+{
491+ struct hl_vm_phys_pg_pack *phys_pg_pack;
447492 struct hl_ctx *ctx = hdev->compute_ctx;
448− int rc = 0;
493+ struct hl_vm_hash_node *hnode;
494+ struct hl_userptr *userptr;
495+ enum vm_type_t *vm_type;
496+ bool valid = false;
497+ u64 end_address;
498+ u32 range_size;
499+ int i, rc = 0;
449500
450501 if (!ctx) {
451502 dev_err(hdev->dev, "no ctx available\n");
452503 return -EINVAL;
453504 }
454505
506+ /* Verify address is mapped */
507+ mutex_lock(&ctx->mem_hash_lock);
508+ hash_for_each(ctx->mem_hash, i, hnode, node) {
509+ vm_type = hnode->ptr;
510+
511+ if (*vm_type == VM_TYPE_USERPTR) {
512+ userptr = hnode->ptr;
513+ range_size = userptr->size;
514+ } else {
515+ phys_pg_pack = hnode->ptr;
516+ range_size = phys_pg_pack->total_size;
517+ }
518+
519+ end_address = virt_addr + size;
520+ if ((virt_addr >= hnode->vaddr) &&
521+ (end_address <= hnode->vaddr + range_size)) {
522+ valid = true;
523+ break;
524+ }
525+ }
526+ mutex_unlock(&ctx->mem_hash_lock);
527+
528+ if (!valid) {
529+ dev_err(hdev->dev,
530+ "virt addr 0x%llx is not mapped\n",
531+ virt_addr);
532+ return -EINVAL;
533+ }
534+
455535 rc = hl_mmu_va_to_pa(ctx, virt_addr, phys_addr);
456536 if (rc) {
457− dev_err(hdev->dev, "virt addr 0x%llx is not mapped to phys addr\n",
458− virt_addr);
537+ dev_err(hdev->dev,
538+ "virt addr 0x%llx is not mapped to phys addr\n",
539+ virt_addr);
459540 rc = -EINVAL;
460541 }
461542
@@ -467,10 +548,11 @@ static ssize_t hl_data_read32(struct file *f, char __user *buf, · 5 unmodified lines
467548 {
468549 struct hl_dbg_device_entry *entry = file_inode(f)->i_private;
469550 struct hl_device *hdev = entry->hdev;
470− char tmp_buf[32];
471551 u64 addr = entry->addr;
472− u32 val;
552+ bool user_address;
553+ char tmp_buf[32];
473554 ssize_t rc;
555+ u32 val;
474556
475557 if (atomic_read(&hdev->in_reset)) {
476558 dev_warn_ratelimited(hdev->dev, "Can't read during reset\n");
@@ -480,13 +562,14 @@ static ssize_t hl_data_read32(struct file *f, char __user *buf, · 3 unmodified lines
480562 if (*ppos)
481563 return 0;
482564
483− if (hl_is_device_va(hdev, addr)) {
484− rc = device_va_to_pa(hdev, addr, &addr);
565+ user_address = hl_is_device_va(hdev, addr);
566+ if (user_address) {
567+ rc = device_va_to_pa(hdev, addr, sizeof(val), &addr);
485568 if (rc)
486569 return rc;
487570 }
488571
489− rc = hdev->asic_funcs->debugfs_read32(hdev, addr, &val);
572+ rc = hdev->asic_funcs->debugfs_read32(hdev, addr, user_address, &val);
490573 if (rc) {
491574 dev_err(hdev->dev, "Failed to read from 0x%010llx\n", addr);
492575 return rc;
@@ -503,6 +586,7 @@ static ssize_t hl_data_write32(struct file *f, const char __user *buf, · 10 unmodified lines
503586 struct hl_dbg_device_entry *entry = file_inode(f)->i_private;
504587 struct hl_device *hdev = entry->hdev;
505588 u64 addr = entry->addr;
589+ bool user_address;
506590 u32 value;
507591 ssize_t rc;
508592
@@ -515,13 +599,14 @@ static ssize_t hl_data_write32(struct file *f, const char __user *buf, · 6 unmodified lines
515599 if (rc)
516600 return rc;
517601
518− if (hl_is_device_va(hdev, addr)) {
519− rc = device_va_to_pa(hdev, addr, &addr);
602+ user_address = hl_is_device_va(hdev, addr);
603+ if (user_address) {
604+ rc = device_va_to_pa(hdev, addr, sizeof(value), &addr);
520605 if (rc)
521606 return rc;
522607 }
523608
524− rc = hdev->asic_funcs->debugfs_write32(hdev, addr, value);
609+ rc = hdev->asic_funcs->debugfs_write32(hdev, addr, user_address, value);
525610 if (rc) {
526611 dev_err(hdev->dev, "Failed to write 0x%08x to 0x%010llx\n",
527612 value, addr);
@@ -536,21 +621,28 @@ static ssize_t hl_data_read64(struct file *f, char __user *buf, · 8 unmodified lines
536621 {
537622 struct hl_dbg_device_entry *entry = file_inode(f)->i_private;
538623 struct hl_device *hdev = entry->hdev;
539− char tmp_buf[32];
540624 u64 addr = entry->addr;
541− u64 val;
625+ bool user_address;
626+ char tmp_buf[32];
542627 ssize_t rc;
628+ u64 val;
629+
630+ if (atomic_read(&hdev->in_reset)) {
631+ dev_warn_ratelimited(hdev->dev, "Can't read during reset\n");
632+ return 0;
633+ }
543634
544635 if (*ppos)
545636 return 0;
546637
547− if (hl_is_device_va(hdev, addr)) {
548− rc = device_va_to_pa(hdev, addr, &addr);
638+ user_address = hl_is_device_va(hdev, addr);
639+ if (user_address) {
640+ rc = device_va_to_pa(hdev, addr, sizeof(val), &addr);
549641 if (rc)
550642 return rc;
551643 }
552644
553− rc = hdev->asic_funcs->debugfs_read64(hdev, addr, &val);
645+ rc = hdev->asic_funcs->debugfs_read64(hdev, addr, user_address, &val);
554646 if (rc) {
555647 dev_err(hdev->dev, "Failed to read from 0x%010llx\n", addr);
556648 return rc;
@@ -567,20 +659,27 @@ static ssize_t hl_data_write64(struct file *f, const char __user *buf, · 10 unmodified lines
567659 struct hl_dbg_device_entry *entry = file_inode(f)->i_private;
568660 struct hl_device *hdev = entry->hdev;
569661 u64 addr = entry->addr;
662+ bool user_address;
570663 u64 value;
571664 ssize_t rc;
572665
666+ if (atomic_read(&hdev->in_reset)) {
667+ dev_warn_ratelimited(hdev->dev, "Can't write during reset\n");
668+ return 0;
669+ }
670+
573671 rc = kstrtoull_from_user(buf, count, 16, &value);
574672 if (rc)
575673 return rc;
576674
577− if (hl_is_device_va(hdev, addr)) {
578− rc = device_va_to_pa(hdev, addr, &addr);
675+ user_address = hl_is_device_va(hdev, addr);
676+ if (user_address) {
677+ rc = device_va_to_pa(hdev, addr, sizeof(value), &addr);
579678 if (rc)
580679 return rc;
581680 }
582681
583− rc = hdev->asic_funcs->debugfs_write64(hdev, addr, value);
682+ rc = hdev->asic_funcs->debugfs_write64(hdev, addr, user_address, value);
584683 if (rc) {
585684 dev_err(hdev->dev, "Failed to write 0x%016llx to 0x%010llx\n",
586685 value, addr);
@@ -590,6 +689,63 @@ static ssize_t hl_data_write64(struct file *f, const char __user *buf, · 3 unmodified lines
590689 return count;
591690 }
592691
692+static ssize_t hl_dma_size_write(struct file *f, const char __user *buf,
693+ size_t count, loff_t *ppos)
694+{
695+ struct hl_dbg_device_entry *entry = file_inode(f)->i_private;
696+ struct hl_device *hdev = entry->hdev;
697+ u64 addr = entry->addr;
698+ ssize_t rc;
699+ u32 size;
700+
701+ if (atomic_read(&hdev->in_reset)) {
702+ dev_warn_ratelimited(hdev->dev, "Can't DMA during reset\n");
703+ return 0;
704+ }
705+ rc = kstrtouint_from_user(buf, count, 16, &size);
706+ if (rc)
707+ return rc;
708+
709+ if (!size) {
710+ dev_err(hdev->dev, "DMA read failed. size can't be 0\n");
711+ return -EINVAL;
712+ }
713+
714+ if (size > SZ_128M) {
715+ dev_err(hdev->dev,
716+ "DMA read failed. size can't be larger than 128MB\n");
717+ return -EINVAL;
718+ }
719+
720+ if (!hl_is_device_internal_memory_va(hdev, addr, size)) {
721+ dev_err(hdev->dev,
722+ "DMA read failed. Invalid 0x%010llx + 0x%08x\n",
723+ addr, size);
724+ return -EINVAL;
725+ }
726+
727+ /* Free the previous allocation, if there was any */
728+ entry->blob_desc.size = 0;
729+ vfree(entry->blob_desc.data);
730+
731+ entry->blob_desc.data = vmalloc(size);
732+ if (!entry->blob_desc.data)
733+ return -ENOMEM;
734+
735+ rc = hdev->asic_funcs->debugfs_read_dma(hdev, addr, size,
736+ entry->blob_desc.data);
737+ if (rc) {
738+ dev_err(hdev->dev, "Failed to DMA from 0x%010llx\n", addr);
739+ vfree(entry->blob_desc.data);
740+ entry->blob_desc.data = NULL;
741+ return -EIO;
742+ }
743+
744+ entry->blob_desc.size = size;
745+
746+ return count;
747+}
748+
593749 static ssize_t hl_get_power_state(struct file *f, char __user *buf,
594750 size_t count, loff_t *ppos)
595751 {
@@ -871,7 +1027,7 @@ static ssize_t hl_stop_on_err_write(struct file *f, const char __user *buf, · 275 unmodified lines
8711027
8721028 hdev->stop_on_err = value ? 1 : 0;
8731029
874− hl_device_reset(hdev, false, false);
1030+ hl_device_reset(hdev, 0);
8751031
8761032 return count;
8771033 }
@@ -899,6 +1055,11 @@ static const struct file_operations hl_data64b_fops = { · 21 unmodified lines
8991055 .write = hl_data_write64
9001056 };
9011057
1058+static const struct file_operations hl_dma_size_fops = {
1059+ .owner = THIS_MODULE,
1060+ .write = hl_dma_size_write
1061+};
1062+
9021063 static const struct file_operations hl_i2c_data_fops = {
9031064 .owner = THIS_MODULE,
9041065 .read = hl_i2c_data_read,
@@ -1001,6 +1162,9 @@ void hl_debugfs_add_device(struct hl_device *hdev) · 96 unmodified lines
10011162 if (!dev_entry->entry_arr)
10021163 return;
10031164
1165+ dev_entry->blob_desc.size = 0;
1166+ dev_entry->blob_desc.data = NULL;
1167+
10041168 INIT_LIST_HEAD(&dev_entry->file_list);
10051169 INIT_LIST_HEAD(&dev_entry->cb_list);
10061170 INIT_LIST_HEAD(&dev_entry->cs_list);
@@ -1103,6 +1267,17 @@ void hl_debugfs_add_device(struct hl_device *hdev) · 96 unmodified lines
11031267 dev_entry,
11041268 &hl_security_violations_fops);
11051269
1270+ debugfs_create_file("dma_size",
1271+ 0200,
1272+ dev_entry->root,
1273+ dev_entry,
1274+ &hl_dma_size_fops);
1275+
1276+ debugfs_create_blob("data_dma",
1277+ 0400,
1278+ dev_entry->root,
1279+ &dev_entry->blob_desc);
1280+
11061281 for (i = 0, entry = dev_entry->entry_arr ; i < count ; i++, entry++) {
11071282 debugfs_create_file(hl_debugfs_list[i].name,
11081283 0444,
@@ -1121,6 +1296,9 @@ void hl_debugfs_remove_device(struct hl_device *hdev) · 12 unmodified lines
11211296 debugfs_remove_recursive(entry->root);
11221297
11231298 mutex_destroy(&entry->file_mutex);
1299+
1300+ vfree(entry->blob_desc.data);
1301+
11241302 kfree(entry->entry_arr);
11251303 }
11261304
drivers/misc/habanalabs/common/device.c+94−127
@@ -70,6 +70,9 @@ static void hpriv_release(struct kref *ref) · 69 unmodified lines
7070 mutex_unlock(&hdev->fpriv_list_lock);
7171
7272 kfree(hpriv);
73+
74+ if (hdev->reset_upon_device_release)
75+ hl_device_reset(hdev, 0);
7376 }
7477
7578 void hl_hpriv_get(struct hl_fpriv *hpriv)
@@ -77,9 +80,9 @@ void hl_hpriv_get(struct hl_fpriv *hpriv) · 1 unmodified line
7780 kref_get(&hpriv->refcount);
7881 }
7982
80−void hl_hpriv_put(struct hl_fpriv *hpriv)
83+int hl_hpriv_put(struct hl_fpriv *hpriv)
8184 {
82− kref_put(&hpriv->refcount, hpriv_release);
85+ return kref_put(&hpriv->refcount, hpriv_release);
8386 }
8487
8588 /*
@@ -103,10 +106,17 @@ static int hl_device_release(struct inode *inode, struct file *filp) · 17 unmodified lines
103106 return 0;
104107 }
105108
106− hl_cb_mgr_fini(hpriv->hdev, &hpriv->cb_mgr);
107− hl_ctx_mgr_fini(hpriv->hdev, &hpriv->ctx_mgr);
109+ /* Each pending user interrupt holds the user's context, hence we
110+ * must release them all before calling hl_ctx_mgr_fini().
111+ */
112+ hl_release_pending_user_interrupts(hpriv->hdev);
113+
114+ hl_cb_mgr_fini(hdev, &hpriv->cb_mgr);
115+ hl_ctx_mgr_fini(hdev, &hpriv->ctx_mgr);
108116
109− hl_hpriv_put(hpriv);
117+ if (!hl_hpriv_put(hpriv))
118+ dev_warn(hdev->dev,
119+ "Device is still in use because there are live CS and/or memory mappings\n");
110120
111121 return 0;
112122 }
@@ -283,7 +293,7 @@ static void device_hard_reset_pending(struct work_struct *work) · 170 unmodified lines
283293 struct hl_device *hdev = device_reset_work->hdev;
284294 int rc;
285295
286− rc = hl_device_reset(hdev, true, true);
296+ rc = hl_device_reset(hdev, HL_RESET_HARD | HL_RESET_FROM_RESET_THREAD);
287297 if ((rc == -EBUSY) && !hdev->device_fini_pending) {
288298 dev_info(hdev->dev,
289299 "Could not reset device. will try again in %u seconds",
@@ -311,11 +321,15 @@ static int device_early_init(struct hl_device *hdev) · 21 unmodified lines
311321 switch (hdev->asic_type) {
312322 case ASIC_GOYA:
313323 goya_set_asic_funcs(hdev);
314− strlcpy(hdev->asic_name, "GOYA", sizeof(hdev->asic_name));
324+ strscpy(hdev->asic_name, "GOYA", sizeof(hdev->asic_name));
315325 break;
316326 case ASIC_GAUDI:
317327 gaudi_set_asic_funcs(hdev);
318− sprintf(hdev->asic_name, "GAUDI");
328+ strscpy(hdev->asic_name, "GAUDI", sizeof(hdev->asic_name));
329+ break;
330+ case ASIC_GAUDI_SEC:
331+ gaudi_set_asic_funcs(hdev);
332+ strscpy(hdev->asic_name, "GAUDI SEC", sizeof(hdev->asic_name));
319333 break;
320334 default:
321335 dev_err(hdev->dev, "Unrecognized ASIC type %d\n",
@@ -334,7 +348,7 @@ static int device_early_init(struct hl_device *hdev) · 12 unmodified lines
334348 if (hdev->asic_prop.completion_queues_count) {
335349 hdev->cq_wq = kcalloc(hdev->asic_prop.completion_queues_count,
336350 sizeof(*hdev->cq_wq),
337− GFP_ATOMIC);
351+ GFP_KERNEL);
338352 if (!hdev->cq_wq) {
339353 rc = -ENOMEM;
340354 goto asid_fini;
@@ -358,24 +372,24 @@ static int device_early_init(struct hl_device *hdev) · 17 unmodified lines
358372 goto free_cq_wq;
359373 }
360374
361− hdev->hl_chip_info = kzalloc(sizeof(struct hwmon_chip_info),
362− GFP_KERNEL);
363− if (!hdev->hl_chip_info) {
375+ hdev->sob_reset_wq = alloc_workqueue("hl-sob-reset", WQ_UNBOUND, 0);
376+ if (!hdev->sob_reset_wq) {
377+ dev_err(hdev->dev,
378+ "Failed to allocate SOB reset workqueue\n");
364379 rc = -ENOMEM;
365380 goto free_eq_wq;
366381 }
367382
368− hdev->idle_busy_ts_arr = kmalloc_array(HL_IDLE_BUSY_TS_ARR_SIZE,
369− sizeof(struct hl_device_idle_busy_ts),
370− (GFP_KERNEL | __GFP_ZERO));
371− if (!hdev->idle_busy_ts_arr) {
383+ hdev->hl_chip_info = kzalloc(sizeof(struct hwmon_chip_info),
384+ GFP_KERNEL);
385+ if (!hdev->hl_chip_info) {
372386 rc = -ENOMEM;
373− goto free_chip_info;
387+ goto free_sob_reset_wq;
374388 }
375389
376390 rc = hl_mmu_if_set_funcs(hdev);
377391 if (rc)
378− goto free_idle_busy_ts_arr;
392+ goto free_chip_info;
379393
380394 hl_cb_mgr_init(&hdev->kernel_cb_mgr);
381395
@@ -404,10 +418,10 @@ static int device_early_init(struct hl_device *hdev) · 22 unmodified lines
404418
405419 free_cb_mgr:
406420 hl_cb_mgr_fini(hdev, &hdev->kernel_cb_mgr);
407−free_idle_busy_ts_arr:
408− kfree(hdev->idle_busy_ts_arr);
409421 free_chip_info:
410422 kfree(hdev->hl_chip_info);
423+free_sob_reset_wq:
424+ destroy_workqueue(hdev->sob_reset_wq);
411425 free_eq_wq:
412426 destroy_workqueue(hdev->eq_wq);
413427 free_cq_wq:
@@ -441,9 +455,9 @@ static void device_early_fini(struct hl_device *hdev) · 27 unmodified lines
441455
442456 hl_cb_mgr_fini(hdev, &hdev->kernel_cb_mgr);
443457
444− kfree(hdev->idle_busy_ts_arr);
445458 kfree(hdev->hl_chip_info);
446459
460+ destroy_workqueue(hdev->sob_reset_wq);
447461 destroy_workqueue(hdev->eq_wq);
448462 destroy_workqueue(hdev->device_reset_work.wq);
449463
@@ -485,7 +499,7 @@ static void hl_device_heartbeat(struct work_struct *work) · 35 unmodified lines
485499 goto reschedule;
486500
487501 dev_err(hdev->dev, "Device heartbeat failed!\n");
488− hl_device_reset(hdev, true, false);
502+ hl_device_reset(hdev, HL_RESET_HARD | HL_RESET_HEARTBEAT);
489503
490504 return;
491505
@@ -561,100 +575,24 @@ static void device_late_fini(struct hl_device *hdev) · 69 unmodified lines
561575 hdev->late_init_done = false;
562576 }
563577
564−uint32_t hl_device_utilization(struct hl_device *hdev, uint32_t period_ms)
578+int hl_device_utilization(struct hl_device *hdev, u32 *utilization)
565579 {
566− struct hl_device_idle_busy_ts *ts;
567− ktime_t zero_ktime, curr = ktime_get();
568− u32 overlap_cnt = 0, last_index = hdev->idle_busy_ts_idx;
569− s64 period_us, last_start_us, last_end_us, last_busy_time_us,
570− total_busy_time_us = 0, total_busy_time_ms;
571−
572− zero_ktime = ktime_set(0, 0);
573− period_us = period_ms * USEC_PER_MSEC;
574− ts = &hdev->idle_busy_ts_arr[last_index];
575−
576− /* check case that device is currently in idle */
577− if (!ktime_compare(ts->busy_to_idle_ts, zero_ktime) &&
578− !ktime_compare(ts->idle_to_busy_ts, zero_ktime)) {
579−
580− last_index--;
581− /* Handle case idle_busy_ts_idx was 0 */
582− if (last_index > HL_IDLE_BUSY_TS_ARR_SIZE)
583− last_index = HL_IDLE_BUSY_TS_ARR_SIZE - 1;
584−
585− ts = &hdev->idle_busy_ts_arr[last_index];
586− }
587−
588− while (overlap_cnt < HL_IDLE_BUSY_TS_ARR_SIZE) {
589− /* Check if we are in last sample case. i.e. if the sample
590− * begun before the sampling period. This could be a real
591− * sample or 0 so need to handle both cases
592− */
593− last_start_us = ktime_to_us(
594− ktime_sub(curr, ts->idle_to_busy_ts));
595−
596− if (last_start_us > period_us) {
597−
598− /* First check two cases:
599− * 1. If the device is currently busy
600− * 2. If the device was idle during the whole sampling
601− * period
602− */
603−
604− if (!ktime_compare(ts->busy_to_idle_ts, zero_ktime)) {
605− /* Check if the device is currently busy */
606− if (ktime_compare(ts->idle_to_busy_ts,
607− zero_ktime))
608− return 100;
609−
610− /* We either didn't have any activity or we
611− * reached an entry which is 0. Either way,
612− * exit and return what was accumulated so far
613− */
614− break;
615− }
616−
617− /* If sample has finished, check it is relevant */
618− last_end_us = ktime_to_us(
619− ktime_sub(curr, ts->busy_to_idle_ts));
620−
621− if (last_end_us > period_us)
622− break;
623−
624− /* It is relevant so add it but with adjustment */
625− last_busy_time_us = ktime_to_us(
626− ktime_sub(ts->busy_to_idle_ts,
627− ts->idle_to_busy_ts));
628− total_busy_time_us += last_busy_time_us -
629− (last_start_us - period_us);
630− break;
631− }
632−
633− /* Check if the sample is finished or still open */
634− if (ktime_compare(ts->busy_to_idle_ts, zero_ktime))
635− last_busy_time_us = ktime_to_us(
636− ktime_sub(ts->busy_to_idle_ts,
637− ts->idle_to_busy_ts));
638− else
639− last_busy_time_us = ktime_to_us(
640− ktime_sub(curr, ts->idle_to_busy_ts));
641−
642− total_busy_time_us += last_busy_time_us;
580+ u64 max_power, curr_power, dc_power, dividend;
581+ int rc;
643582
644− last_index--;
645− /* Handle case idle_busy_ts_idx was 0 */
646− if (last_index > HL_IDLE_BUSY_TS_ARR_SIZE)
647− last_index = HL_IDLE_BUSY_TS_ARR_SIZE - 1;
583+ max_power = hdev->asic_prop.max_power_default;
584+ dc_power = hdev->asic_prop.dc_power_default;
585+ rc = hl_fw_cpucp_power_get(hdev, &curr_power);
648586
649− ts = &hdev->idle_busy_ts_arr[last_index];
587+ if (rc)
588+ return rc;
650589
651− overlap_cnt++;
652− }
590+ curr_power = clamp(curr_power, dc_power, max_power);
653591
654− total_busy_time_ms = DIV_ROUND_UP_ULL(total_busy_time_us,
655− USEC_PER_MSEC);
592+ dividend = (curr_power - dc_power) * 100;
593+ *utilization = (u32) div_u64(dividend, (max_power - dc_power));
656594
657− return DIV_ROUND_UP_ULL(total_busy_time_ms * 100, period_ms);
595+ return 0;
658596 }
659597
660598 /*
@@ -809,7 +747,7 @@ int hl_device_resume(struct hl_device *hdev) · 148 unmodified lines
809747 hdev->disabled = false;
810748 atomic_set(&hdev->in_reset, 0);
811749
812− rc = hl_device_reset(hdev, true, false);
750+ rc = hl_device_reset(hdev, HL_RESET_HARD);
813751 if (rc) {
814752 dev_err(hdev->dev, "Failed to reset device during resume\n");
815753 goto disable_device;
@@ -915,9 +853,7 @@ static void device_disable_open_processes(struct hl_device *hdev) · 99 unmodified lines
915853 * hl_device_reset - reset the device
916854 *
917855 * @hdev: pointer to habanalabs device structure
918− * @hard_reset: should we do hard reset to all engines or just reset the
919− * compute/dma engines
920− * @from_hard_reset_thread: is the caller the hard-reset thread
856+ * @flags: reset flags.
921857 *
922858 * Block future CS and wait for pending CS to be enqueued
923859 * Call ASIC H/W fini
@@ -929,9 +865,10 @@ static void device_disable_open_processes(struct hl_device *hdev) · 5 unmodified lines
929865 *
930866 * Returns 0 for success or an error on failure.
931867 */
932−int hl_device_reset(struct hl_device *hdev, bool hard_reset,
933− bool from_hard_reset_thread)
868+int hl_device_reset(struct hl_device *hdev, u32 flags)
934869 {
870+ u64 idle_mask[HL_BUSY_ENGINES_MASK_EXT_SIZE] = {0};
871+ bool hard_reset, from_hard_reset_thread;
935872 int i, rc;
936873
937874 if (!hdev->init_done) {
@@ -940,6 +877,9 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, · 2 unmodified lines
940877 return 0;
941878 }
942879
880+ hard_reset = (flags & HL_RESET_HARD) != 0;
881+ from_hard_reset_thread = (flags & HL_RESET_FROM_RESET_THREAD) != 0;
882+
943883 if ((!hard_reset) && (!hdev->supports_soft_reset)) {
944884 dev_dbg(hdev->dev, "Doing hard-reset instead of soft-reset\n");
945885 hard_reset = true;
@@ -960,7 +900,11 @@ int hl_device_reset(struct hl_device *hdev, bool hard_reset, · 14 unmodified lines
960900 if (rc)
961901 return 0;
962902
963− if (hard_reset) {
903+ /*
904+ * if reset is due to heartbeat, device CPU is no responsive in
905+ * which case no point sending PCI disable message to it
906+ */
907+ if (hard_reset && !(flags & HL_RESET_HEARTBEAT)) {
964908 /* Disable PCI access from device F/W so he won't send
965909 * us additional interrupts. We disable MSI/MSI-X at
966910 * the halt_engines function and we can't have the F/W
@@ -1030,6 +974,11 @@ again: · 63 unmodified lines
1030974 /* Go over all the queues, release all CS and their jobs */
1031975 hl_cs_rollback_all(hdev);
1032976
977+ /* Release all pending user interrupts, each pending user interrupt
978+ * holds a reference to user context
979+ */
980+ hl_release_pending_user_interrupts(hdev);
981+
1033982 kill_processes:
1034983 if (hard_reset) {
1035984 /* Kill processes here after CS rollback. This is because the
@@ -1078,14 +1027,6 @@ kill_processes: · 42 unmodified lines
10781027 for (i = 0 ; i < hdev->asic_prop.completion_queues_count ; i++)
10791028 hl_cq_reset(hdev, &hdev->completion_queue[i]);
10801029
1081− hdev->idle_busy_ts_idx = 0;
1082− hdev->idle_busy_ts_arr[0].busy_to_idle_ts = ktime_set(0, 0);
1083− hdev->idle_busy_ts_arr[0].idle_to_busy_ts = ktime_set(0, 0);
1084−
1085− if (hdev->cs_active_cnt)
1086− dev_crit(hdev->dev, "CS active cnt %d is not 0 during reset\n",
1087− hdev->cs_active_cnt);
1088−
10891030 mutex_lock(&hdev->fpriv_list_lock);
10901031
10911032 /* Make sure the context switch phase will run again */
@@ -1151,6 +1092,16 @@ kill_processes: · 59 unmodified lines
11511092 goto out_err;
11521093 }
11531094
1095+ /* If device is not idle fail the reset process */
1096+ if (!hdev->asic_funcs->is_device_idle(hdev, idle_mask,
1097+ HL_BUSY_ENGINES_MASK_EXT_SIZE, NULL)) {
1098+ dev_err(hdev->dev,
1099+ "device is not idle (mask %#llx %#llx) after reset\n",
1100+ idle_mask[0], idle_mask[1]);
1101+ rc = -EIO;
1102+ goto out_err;
1103+ }
1104+
11541105 /* Check that the communication with the device is working */
11551106 rc = hdev->asic_funcs->test_queues(hdev);
11561107 if (rc) {
@@ -1235,7 +1186,7 @@ out_err: · 78 unmodified lines
12351186 */
12361187 int hl_device_init(struct hl_device *hdev, struct class *hclass)
12371188 {
1238− int i, rc, cq_cnt, cq_ready_cnt;
1189+ int i, rc, cq_cnt, user_interrupt_cnt, cq_ready_cnt;
12391190 char *name;
12401191 bool add_cdev_sysfs_on_err = false;
12411192
@@ -1274,13 +1225,26 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass) · 32 unmodified lines
12741225 if (rc)
12751226 goto free_dev_ctrl;
12761227
1228+ user_interrupt_cnt = hdev->asic_prop.user_interrupt_count;
1229+
1230+ if (user_interrupt_cnt) {
1231+ hdev->user_interrupt = kcalloc(user_interrupt_cnt,
1232+ sizeof(*hdev->user_interrupt),
1233+ GFP_KERNEL);
1234+
1235+ if (!hdev->user_interrupt) {
1236+ rc = -ENOMEM;
1237+ goto early_fini;
1238+ }
1239+ }
1240+
12771241 /*
12781242 * Start calling ASIC initialization. First S/W then H/W and finally
12791243 * late init
12801244 */
12811245 rc = hdev->asic_funcs->sw_init(hdev);
12821246 if (rc)
1283− goto early_fini;
1247+ goto user_interrupts_fini;
12841248
12851249 /*
12861250 * Initialize the H/W queues. Must be done before hw_init, because
@@ -1478,6 +1442,8 @@ hw_queues_destroy: · 191 unmodified lines
14781442 hl_hw_queues_destroy(hdev);
14791443 sw_fini:
14801444 hdev->asic_funcs->sw_fini(hdev);
1445+user_interrupts_fini:
1446+ kfree(hdev->user_interrupt);
14811447 early_fini:
14821448 device_early_fini(hdev);
14831449 free_dev_ctrl:
@@ -1609,6 +1575,7 @@ void hl_device_fini(struct hl_device *hdev) · 125 unmodified lines
16091575 for (i = 0 ; i < hdev->asic_prop.completion_queues_count ; i++)
16101576 hl_cq_fini(hdev, &hdev->completion_queue[i]);
16111577 kfree(hdev->completion_queue);
1578+ kfree(hdev->user_interrupt);
16121579
16131580 hl_hw_queues_destroy(hdev);
16141581
drivers/misc/habanalabs/common/firmware_if.c+216−22
@@ -293,6 +293,7 @@ static int fw_read_errors(struct hl_device *hdev, u32 boot_err0_reg, · 292 unmodified lines
293293 u32 cpu_security_boot_status_reg)
294294 {
295295 u32 err_val, security_val;
296+ bool err_exists = false;
296297
297298 /* Some of the firmware status codes are deprecated in newer f/w
298299 * versions. In those versions, the errors are reported
@@ -307,48 +308,102 @@ static int fw_read_errors(struct hl_device *hdev, u32 boot_err0_reg, · 8 unmodified lines
307308 if (!(err_val & CPU_BOOT_ERR0_ENABLED))
308309 return 0;
309310
310− if (err_val & CPU_BOOT_ERR0_DRAM_INIT_FAIL)
311+ if (err_val & CPU_BOOT_ERR0_DRAM_INIT_FAIL) {
311312 dev_err(hdev->dev,
312313 "Device boot error - DRAM initialization failed\n");
313− if (err_val & CPU_BOOT_ERR0_FIT_CORRUPTED)
314+ err_exists = true;
315+ }
316+
317+ if (err_val & CPU_BOOT_ERR0_FIT_CORRUPTED) {
314318 dev_err(hdev->dev, "Device boot error - FIT image corrupted\n");
315− if (err_val & CPU_BOOT_ERR0_TS_INIT_FAIL)
319+ err_exists = true;
320+ }
321+
322+ if (err_val & CPU_BOOT_ERR0_TS_INIT_FAIL) {
316323 dev_err(hdev->dev,
317324 "Device boot error - Thermal Sensor initialization failed\n");
318− if (err_val & CPU_BOOT_ERR0_DRAM_SKIPPED)
325+ err_exists = true;
326+ }
327+
328+ if (err_val & CPU_BOOT_ERR0_DRAM_SKIPPED) {
319329 dev_warn(hdev->dev,
320330 "Device boot warning - Skipped DRAM initialization\n");
331+ /* This is a warning so we don't want it to disable the
332+ * device
333+ */
334+ err_val &= ~CPU_BOOT_ERR0_DRAM_SKIPPED;
335+ }
321336
322337 if (err_val & CPU_BOOT_ERR0_BMC_WAIT_SKIPPED) {
323− if (hdev->bmc_enable)
324− dev_warn(hdev->dev,
338+ if (hdev->bmc_enable) {
339+ dev_err(hdev->dev,
325340 "Device boot error - Skipped waiting for BMC\n");
326− else
341+ err_exists = true;
342+ } else {
343+ dev_info(hdev->dev,
344+ "Device boot message - Skipped waiting for BMC\n");
345+ /* This is an info so we don't want it to disable the
346+ * device
347+ */
327348 err_val &= ~CPU_BOOT_ERR0_BMC_WAIT_SKIPPED;
349+ }
328350 }
329351
330− if (err_val & CPU_BOOT_ERR0_NIC_DATA_NOT_RDY)
352+ if (err_val & CPU_BOOT_ERR0_NIC_DATA_NOT_RDY) {
331353 dev_err(hdev->dev,
332354 "Device boot error - Serdes data from BMC not available\n");
333− if (err_val & CPU_BOOT_ERR0_NIC_FW_FAIL)
355+ err_exists = true;
356+ }
357+
358+ if (err_val & CPU_BOOT_ERR0_NIC_FW_FAIL) {
334359 dev_err(hdev->dev,
335360 "Device boot error - NIC F/W initialization failed\n");
336− if (err_val & CPU_BOOT_ERR0_SECURITY_NOT_RDY)
361+ err_exists = true;
362+ }
363+
364+ if (err_val & CPU_BOOT_ERR0_SECURITY_NOT_RDY) {
337365 dev_warn(hdev->dev,
338366 "Device boot warning - security not ready\n");
339− if (err_val & CPU_BOOT_ERR0_SECURITY_FAIL)
367+ /* This is a warning so we don't want it to disable the
368+ * device
369+ */
370+ err_val &= ~CPU_BOOT_ERR0_SECURITY_NOT_RDY;
371+ }
372+
373+ if (err_val & CPU_BOOT_ERR0_SECURITY_FAIL) {
340374 dev_err(hdev->dev, "Device boot error - security failure\n");
341− if (err_val & CPU_BOOT_ERR0_EFUSE_FAIL)
375+ err_exists = true;
376+ }
377+
378+ if (err_val & CPU_BOOT_ERR0_EFUSE_FAIL) {
342379 dev_err(hdev->dev, "Device boot error - eFuse failure\n");
343− if (err_val & CPU_BOOT_ERR0_PLL_FAIL)
380+ err_exists = true;
381+ }
382+
383+ if (err_val & CPU_BOOT_ERR0_PLL_FAIL) {
344384 dev_err(hdev->dev, "Device boot error - PLL failure\n");
385+ err_exists = true;
386+ }
387+
388+ if (err_val & CPU_BOOT_ERR0_DEVICE_UNUSABLE_FAIL) {
389+ dev_err(hdev->dev,
390+ "Device boot error - device unusable\n");
391+ err_exists = true;
392+ }
345393
346394 security_val = RREG32(cpu_security_boot_status_reg);
347395 if (security_val & CPU_BOOT_DEV_STS0_ENABLED)
348396 dev_dbg(hdev->dev, "Device security status %#x\n",
349397 security_val);
350398
351− if (err_val & ~CPU_BOOT_ERR0_ENABLED)
399+ if (!err_exists && (err_val & ~CPU_BOOT_ERR0_ENABLED)) {
400+ dev_err(hdev->dev,
401+ "Device boot error - unknown error 0x%08x\n",
402+ err_val);
403+ err_exists = true;
404+ }
405+
406+ if (err_exists)
352407 return -EIO;
353408
354409 return 0;
@@ -419,6 +474,73 @@ out: · 64 unmodified lines
419474 return rc;
420475 }
421476
477+static int hl_fw_send_msi_info_msg(struct hl_device *hdev)
478+{
479+ struct cpucp_array_data_packet *pkt;
480+ size_t total_pkt_size, data_size;
481+ u64 result;
482+ int rc;
483+
484+ /* skip sending this info for unsupported ASICs */
485+ if (!hdev->asic_funcs->get_msi_info)
486+ return 0;
487+
488+ data_size = CPUCP_NUM_OF_MSI_TYPES * sizeof(u32);
489+ total_pkt_size = sizeof(struct cpucp_array_data_packet) + data_size;
490+
491+ /* data should be aligned to 8 bytes in order to CPU-CP to copy it */
492+ total_pkt_size = (total_pkt_size + 0x7) & ~0x7;
493+
494+ /* total_pkt_size is casted to u16 later on */
495+ if (total_pkt_size > USHRT_MAX) {
496+ dev_err(hdev->dev, "CPUCP array data is too big\n");
497+ return -EINVAL;
498+ }
499+
500+ pkt = kzalloc(total_pkt_size, GFP_KERNEL);
501+ if (!pkt)
502+ return -ENOMEM;
503+
504+ pkt->length = cpu_to_le32(CPUCP_NUM_OF_MSI_TYPES);
505+
506+ hdev->asic_funcs->get_msi_info((u32 *)&pkt->data);
507+
508+ pkt->cpucp_pkt.ctl = cpu_to_le32(CPUCP_PACKET_MSI_INFO_SET <<
509+ CPUCP_PKT_CTL_OPCODE_SHIFT);
510+
511+ rc = hdev->asic_funcs->send_cpu_message(hdev, (u32 *)pkt,
512+ total_pkt_size, 0, &result);
513+
514+ /*
515+ * in case packet result is invalid it means that FW does not support
516+ * this feature and will use default/hard coded MSI values. no reason
517+ * to stop the boot
518+ */
519+ if (rc && result == cpucp_packet_invalid)
520+ rc = 0;
521+
522+ if (rc)
523+ dev_err(hdev->dev, "failed to send CPUCP array data\n");
524+
525+ kfree(pkt);
526+
527+ return rc;
528+}
529+
530+int hl_fw_cpucp_handshake(struct hl_device *hdev,
531+ u32 cpu_security_boot_status_reg,
532+ u32 boot_err0_reg)
533+{
534+ int rc;
535+
536+ rc = hl_fw_cpucp_info_get(hdev, cpu_security_boot_status_reg,
537+ boot_err0_reg);
538+ if (rc)
539+ return rc;
540+
541+ return hl_fw_send_msi_info_msg(hdev);
542+}
543+
422544 int hl_fw_get_eeprom_data(struct hl_device *hdev, void *data, size_t max_size)
423545 {
424546 struct cpucp_packet pkt = {};
@@ -539,18 +661,63 @@ int hl_fw_cpucp_total_energy_get(struct hl_device *hdev, u64 *total_energy) · 114 unmodified lines
539661 return rc;
540662 }
541663
542−int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, u16 pll_index,
664+int get_used_pll_index(struct hl_device *hdev, enum pll_index input_pll_index,
665+ enum pll_index *pll_index)
666+{
667+ struct asic_fixed_properties *prop = &hdev->asic_prop;
668+ u8 pll_byte, pll_bit_off;
669+ bool dynamic_pll;
670+
671+ if (input_pll_index >= PLL_MAX) {
672+ dev_err(hdev->dev, "PLL index %d is out of range\n",
673+ input_pll_index);
674+ return -EINVAL;
675+ }
676+
677+ dynamic_pll = prop->fw_security_status_valid &&
678+ (prop->fw_app_security_map & CPU_BOOT_DEV_STS0_DYN_PLL_EN);
679+
680+ if (!dynamic_pll) {
681+ /*
682+ * in case we are working with legacy FW (each asic has unique
683+ * PLL numbering) extract the legacy numbering
684+ */
685+ *pll_index = hdev->legacy_pll_map[input_pll_index];
686+ return 0;
687+ }
688+
689+ /* PLL map is a u8 array */
690+ pll_byte = prop->cpucp_info.pll_map[input_pll_index >> 3];
691+ pll_bit_off = input_pll_index & 0x7;
692+
693+ if (!(pll_byte & BIT(pll_bit_off))) {
694+ dev_err(hdev->dev, "PLL index %d is not supported\n",
695+ input_pll_index);
696+ return -EINVAL;
697+ }
698+
699+ *pll_index = input_pll_index;
700+
701+ return 0;
702+}
703+
704+int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, enum pll_index pll_index,
543705 u16 *pll_freq_arr)
544706 {
545707 struct cpucp_packet pkt;
708+ enum pll_index used_pll_idx;
546709 u64 result;
547710 int rc;
548711
712+ rc = get_used_pll_index(hdev, pll_index, &used_pll_idx);
713+ if (rc)
714+ return rc;
715+
549716 memset(&pkt, 0, sizeof(pkt));
550717
551718 pkt.ctl = cpu_to_le32(CPUCP_PACKET_PLL_INFO_GET <<
552719 CPUCP_PKT_CTL_OPCODE_SHIFT);
553− pkt.pll_type = __cpu_to_le16(pll_index);
720+ pkt.pll_type = __cpu_to_le16((u16)used_pll_idx);
554721
555722 rc = hdev->asic_funcs->send_cpu_message(hdev, (u32 *) &pkt, sizeof(pkt),
556723 HL_CPUCP_INFO_TIMEOUT_USEC, &result);
@@ -565,6 +732,29 @@ int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, u16 pll_index, · 8 unmodified lines
565732 return rc;
566733 }
567734
735+int hl_fw_cpucp_power_get(struct hl_device *hdev, u64 *power)
736+{
737+ struct cpucp_packet pkt;
738+ u64 result;
739+ int rc;
740+
741+ memset(&pkt, 0, sizeof(pkt));
742+
743+ pkt.ctl = cpu_to_le32(CPUCP_PACKET_POWER_GET <<
744+ CPUCP_PKT_CTL_OPCODE_SHIFT);
745+
746+ rc = hdev->asic_funcs->send_cpu_message(hdev, (u32 *) &pkt, sizeof(pkt),
747+ HL_CPUCP_INFO_TIMEOUT_USEC, &result);
748+ if (rc) {
749+ dev_err(hdev->dev, "Failed to read power, error %d\n", rc);
750+ return rc;
751+ }
752+
753+ *power = result;
754+
755+ return rc;
756+}
757+
568758 static void detect_cpu_boot_status(struct hl_device *hdev, u32 status)
569759 {
570760 /* Some of the status codes below are deprecated in newer f/w
@@ -623,7 +813,11 @@ int hl_fw_read_preboot_status(struct hl_device *hdev, u32 cpu_boot_status_reg, · 52 unmodified lines
623813 u32 status, security_status;
624814 int rc;
625815
626− if (!hdev->cpu_enable)
816+ /* pldm was added for cases in which we use preboot on pldm and want
817+ * to load boot fit, but we can't wait for preboot because it runs
818+ * very slowly
819+ */
820+ if (!(hdev->fw_components & FW_TYPE_PREBOOT_CPU) || hdev->pldm)
627821 return 0;
628822
629823 /* Need to check two possible scenarios:
@@ -677,16 +871,16 @@ int hl_fw_read_preboot_status(struct hl_device *hdev, u32 cpu_boot_status_reg, · 47 unmodified lines
677871 if (security_status & CPU_BOOT_DEV_STS0_ENABLED) {
678872 prop->fw_security_status_valid = 1;
679873
874+ /* FW security should be derived from PCI ID, we keep this
875+ * check for backward compatibility
876+ */
680877 if (security_status & CPU_BOOT_DEV_STS0_SECURITY_EN)
681878 prop->fw_security_disabled = false;
682− else
683− prop->fw_security_disabled = true;
684879
685880 if (security_status & CPU_BOOT_DEV_STS0_FW_HARD_RST_EN)
686881 prop->hard_reset_done_by_fw = true;
687882 } else {
688883 prop->fw_security_status_valid = 0;
689− prop->fw_security_disabled = true;
690884 }
691885
692886 dev_dbg(hdev->dev, "Firmware preboot security status %#x\n",
@@ -710,7 +904,7 @@ int hl_fw_init_cpu(struct hl_device *hdev, u32 cpu_boot_status_reg, · 17 unmodified lines
710904 u32 status;
711905 int rc;
712906
713− if (!(hdev->fw_loading & FW_TYPE_BOOT_CPU))
907+ if (!(hdev->fw_components & FW_TYPE_BOOT_CPU))
714908 return 0;
715909
716910 dev_info(hdev->dev, "Going to wait for device boot (up to %lds)\n",
@@ -801,7 +995,7 @@ int hl_fw_init_cpu(struct hl_device *hdev, u32 cpu_boot_status_reg, · 84 unmodified lines
801995 goto out;
802996 }
803997
804− if (!(hdev->fw_loading & FW_TYPE_LINUX)) {
998+ if (!(hdev->fw_components & FW_TYPE_LINUX)) {
805999 dev_info(hdev->dev, "Skip loading Linux F/W\n");
8061000 goto out;
8071001 }
drivers/misc/habanalabs/common/habanalabs.h+143−41
@@ -19,6 +19,7 @@ · 18 unmodified lines
1919 #include <linux/dma-direction.h>
2020 #include <linux/scatterlist.h>
2121 #include <linux/hashtable.h>
22+#include <linux/debugfs.h>
2223 #include <linux/bitfield.h>
2324 #include <linux/genalloc.h>
2425 #include <linux/sched/signal.h>
@@ -61,7 +62,7 @@ · 36 unmodified lines
6162
6263 #define HL_SIM_MAX_TIMEOUT_US 10000000 /* 10s */
6364
64−#define HL_IDLE_BUSY_TS_ARR_SIZE 4096
65+#define HL_COMMON_USER_INTERRUPT_ID 0xFFF
6566
6667 /* Memory */
6768 #define MEM_HASH_TABLE_BITS 7 /* 1 << 7 buckets */
@@ -102,6 +103,23 @@ enum hl_mmu_page_table_location { · 34 unmodified lines
102103
103104 #define HL_MAX_DCORES 4
104105
106+/*
107+ * Reset Flags
108+ *
109+ * - HL_RESET_HARD
110+ * If set do hard reset to all engines. If not set reset just
111+ * compute/DMA engines.
112+ *
113+ * - HL_RESET_FROM_RESET_THREAD
114+ * Set if the caller is the hard-reset thread
115+ *
116+ * - HL_RESET_HEARTBEAT
117+ * Set if reset is due to heartbeat
118+ */
119+#define HL_RESET_HARD (1 << 0)
120+#define HL_RESET_FROM_RESET_THREAD (1 << 1)
121+#define HL_RESET_HEARTBEAT (1 << 2)
122+
105123 #define HL_MAX_SOBS_PER_MONITOR 8
106124
107125 /**
@@ -169,15 +187,19 @@ enum hl_fw_component { · 61 unmodified lines
169187 };
170188
171189 /**
172− * enum hl_fw_types - F/W types to load
190+ * enum hl_fw_types - F/W types present in the system
173191 * @FW_TYPE_LINUX: Linux image for device CPU
174192 * @FW_TYPE_BOOT_CPU: Boot image for device CPU
193+ * @FW_TYPE_PREBOOT_CPU: Indicates pre-loaded CPUs are present in the system
194+ * (preboot, ppboot etc...)
175195 * @FW_TYPE_ALL_TYPES: Mask for all types
176196 */
177197 enum hl_fw_types {
178198 FW_TYPE_LINUX = 0x1,
179199 FW_TYPE_BOOT_CPU = 0x2,
180− FW_TYPE_ALL_TYPES = (FW_TYPE_LINUX | FW_TYPE_BOOT_CPU)
200+ FW_TYPE_PREBOOT_CPU = 0x4,
201+ FW_TYPE_ALL_TYPES =
202+ (FW_TYPE_LINUX | FW_TYPE_BOOT_CPU | FW_TYPE_PREBOOT_CPU)
181203 };
182204
183205 /**
@@ -368,6 +390,7 @@ struct hl_mmu_properties { · 184 unmodified lines
368390 * @dram_size: DRAM total size.
369391 * @dram_pci_bar_size: size of PCI bar towards DRAM.
370392 * @max_power_default: max power of the device after reset
393+ * @dc_power_default: power consumed by the device in mode idle.
371394 * @dram_size_for_default_page_mapping: DRAM size needed to map to avoid page
372395 * fault.
373396 * @pcie_dbi_base_address: Base address of the PCIE_DBI block.
@@ -412,6 +435,7 @@ struct hl_mmu_properties { · 38 unmodified lines
412435 * @first_available_user_msix_interrupt: first available msix interrupt
413436 * reserved for the user
414437 * @first_available_cq: first available CQ for the user.
438+ * @user_interrupt_count: number of user interrupts.
415439 * @tpc_enabled_mask: which TPCs are enabled.
416440 * @completion_queues_count: number of completion queues.
417441 * @fw_security_disabled: true if security measures are disabled in firmware,
@@ -421,6 +445,7 @@ struct hl_mmu_properties { · 3 unmodified lines
421445 * @dram_supports_virtual_memory: is there an MMU towards the DRAM
422446 * @hard_reset_done_by_fw: true if firmware is handling hard reset flow
423447 * @num_functional_hbms: number of functional HBMs in each DCORE.
448+ * @iatu_done_by_fw: true if iATU configuration is being done by FW.
424449 */
425450 struct asic_fixed_properties {
426451 struct hw_queue_properties *hw_queues_props;
@@ -439,6 +464,7 @@ struct asic_fixed_properties { · 12 unmodified lines
439464 u64 dram_size;
440465 u64 dram_pci_bar_size;
441466 u64 max_power_default;
467+ u64 dc_power_default;
442468 u64 dram_size_for_default_page_mapping;
443469 u64 pcie_dbi_base_address;
444470 u64 pcie_aux_dbi_reg_addr;
@@ -475,6 +501,7 @@ struct asic_fixed_properties { · 30 unmodified lines
475501 u16 first_available_user_mon[HL_MAX_DCORES];
476502 u16 first_available_user_msix_interrupt;
477503 u16 first_available_cq[HL_MAX_DCORES];
504+ u16 user_interrupt_count;
478505 u8 tpc_enabled_mask;
479506 u8 completion_queues_count;
480507 u8 fw_security_disabled;
@@ -482,6 +509,7 @@ struct asic_fixed_properties { · 1 unmodified line
482509 u8 dram_supports_virtual_memory;
483510 u8 hard_reset_done_by_fw;
484511 u8 num_functional_hbms;
512+ u8 iatu_done_by_fw;
485513 };
486514
487515 /**
@@ -503,6 +531,7 @@ struct hl_fence { · 15 unmodified lines
503531
504532 /**
505533 * struct hl_cs_compl - command submission completion object.
534+ * @sob_reset_work: workqueue object to run SOB reset flow.
506535 * @base_fence: hl fence object.
507536 * @lock: spinlock to protect fence.
508537 * @hdev: habanalabs device structure.
@@ -513,6 +542,7 @@ struct hl_fence { · 4 unmodified lines
513542 * @sob_group: the SOB group that is used in this collective wait CS.
514543 */
515544 struct hl_cs_compl {
545+ struct work_struct sob_reset_work;
516546 struct hl_fence base_fence;
517547 spinlock_t lock;
518548 struct hl_device *hdev;
@@ -689,6 +719,31 @@ struct hl_cq { · 170 unmodified lines
689719 atomic_t free_slots_cnt;
690720 };
691721
722+/**
723+ * struct hl_user_interrupt - holds user interrupt information
724+ * @hdev: pointer to the device structure
725+ * @wait_list_head: head to the list of user threads pending on this interrupt
726+ * @wait_list_lock: protects wait_list_head
727+ * @interrupt_id: msix interrupt id
728+ */
729+struct hl_user_interrupt {
730+ struct hl_device *hdev;
731+ struct list_head wait_list_head;
732+ spinlock_t wait_list_lock;
733+ u32 interrupt_id;
734+};
735+
736+/**
737+ * struct hl_user_pending_interrupt - holds a context to a user thread
738+ * pending on an interrupt
739+ * @wait_list_node: node in the list of user threads pending on an interrupt
740+ * @fence: hl fence object for interrupt completion
741+ */
742+struct hl_user_pending_interrupt {
743+ struct list_head wait_list_node;
744+ struct hl_fence fence;
745+};
746+
692747 /**
693748 * struct hl_eq - describes the event queue (single one per device)
694749 * @hdev: pointer to the device structure
@@ -713,11 +768,13 @@ struct hl_eq { · 18 unmodified lines
713768 * @ASIC_INVALID: Invalid ASIC type.
714769 * @ASIC_GOYA: Goya device.
715770 * @ASIC_GAUDI: Gaudi device.
771+ * @ASIC_GAUDI_SEC: Gaudi secured device (HL-2000).
716772 */
717773 enum hl_asic_type {
718774 ASIC_INVALID,
719775 ASIC_GOYA,
720− ASIC_GAUDI
776+ ASIC_GAUDI,
777+ ASIC_GAUDI_SEC
721778 };
722779
723780 struct hl_cs_parser;
@@ -802,8 +859,12 @@ enum div_select_defs { · 78 unmodified lines
802859 * @update_eq_ci: update event queue CI.
803860 * @context_switch: called upon ASID context switch.
804861 * @restore_phase_topology: clear all SOBs amd MONs.
805− * @debugfs_read32: debug interface for reading u32 from DRAM/SRAM.
806− * @debugfs_write32: debug interface for writing u32 to DRAM/SRAM.
862+ * @debugfs_read32: debug interface for reading u32 from DRAM/SRAM/Host memory.
863+ * @debugfs_write32: debug interface for writing u32 to DRAM/SRAM/Host memory.
864+ * @debugfs_read64: debug interface for reading u64 from DRAM/SRAM/Host memory.
865+ * @debugfs_write64: debug interface for writing u64 to DRAM/SRAM/Host memory.
866+ * @debugfs_read_dma: debug interface for reading up to 2MB from the device's
867+ * internal memory via DMA engine.
807868 * @add_device_attr: add ASIC specific device attributes.
808869 * @handle_eqe: handle event queue entry (IRQ) from CPU-CP.
809870 * @set_pll_profile: change PLL profile (manual/automatic).
@@ -919,10 +980,16 @@ struct hl_asic_funcs { · 109 unmodified lines
919980 void (*update_eq_ci)(struct hl_device *hdev, u32 val);
920981 int (*context_switch)(struct hl_device *hdev, u32 asid);
921982 void (*restore_phase_topology)(struct hl_device *hdev);
922− int (*debugfs_read32)(struct hl_device *hdev, u64 addr, u32 *val);
923− int (*debugfs_write32)(struct hl_device *hdev, u64 addr, u32 val);
924− int (*debugfs_read64)(struct hl_device *hdev, u64 addr, u64 *val);
925− int (*debugfs_write64)(struct hl_device *hdev, u64 addr, u64 val);
983+ int (*debugfs_read32)(struct hl_device *hdev, u64 addr,
984+ bool user_address, u32 *val);
985+ int (*debugfs_write32)(struct hl_device *hdev, u64 addr,
986+ bool user_address, u32 val);
987+ int (*debugfs_read64)(struct hl_device *hdev, u64 addr,
988+ bool user_address, u64 *val);
989+ int (*debugfs_write64)(struct hl_device *hdev, u64 addr,
990+ bool user_address, u64 val);
991+ int (*debugfs_read_dma)(struct hl_device *hdev, u64 addr, u32 size,
992+ void *blob_addr);
926993 void (*add_device_attr)(struct hl_device *hdev,
927994 struct attribute_group *dev_attr_grp);
928995 void (*handle_eqe)(struct hl_device *hdev,
@@ -986,6 +1053,7 @@ struct hl_asic_funcs { · 57 unmodified lines
9861053 int (*hw_block_mmap)(struct hl_device *hdev, struct vm_area_struct *vma,
9871054 u32 block_id, u32 block_size);
9881055 void (*enable_events_from_fw)(struct hl_device *hdev);
1056+ void (*get_msi_info)(u32 *table);
9891057 };
9901058
9911059
@@ -1070,9 +1138,11 @@ struct hl_pending_cb { · 78 unmodified lines
10701138 * @mem_hash_lock: protects the mem_hash.
10711139 * @mmu_lock: protects the MMU page tables. Any change to the PGT, modifying the
10721140 * MMU hash or walking the PGT requires talking this lock.
1141+ * @hw_block_list_lock: protects the HW block memory list.
10731142 * @debugfs_list: node in debugfs list of contexts.
10741143 * pending_cb_list: list of pending command buffers waiting to be sent upon
10751144 * next user command submission context.
1145+ * @hw_block_mem_list: list of HW block virtual mapped addresses.
10761146 * @cs_counters: context command submission counters.
10771147 * @cb_va_pool: device VA pool for command buffers which are mapped to the
10781148 * device's MMU.
@@ -1109,8 +1179,10 @@ struct hl_ctx { · 30 unmodified lines
11091179 struct hl_va_range *va_range[HL_VA_RANGE_TYPE_MAX];
11101180 struct mutex mem_hash_lock;
11111181 struct mutex mmu_lock;
1182+ struct mutex hw_block_list_lock;
11121183 struct list_head debugfs_list;
11131184 struct list_head pending_cb_list;
1185+ struct list_head hw_block_mem_list;
11141186 struct hl_cs_counters_atomic cs_counters;
11151187 struct gen_pool *cb_va_pool;
11161188 u64 cs_sequence;
@@ -1185,6 +1257,7 @@ struct hl_userptr { · 68 unmodified lines
11851257 * @sequence: the sequence number of this CS.
11861258 * @staged_sequence: the sequence of the staged submission this CS is part of,
11871259 * relevant only if staged_cs is set.
1260+ * @timeout_jiffies: cs timeout in jiffies.
11881261 * @type: CS_TYPE_*.
11891262 * @submitted: true if CS was submitted to H/W.
11901263 * @completed: true if CS was completed by device.
@@ -1213,6 +1286,7 @@ struct hl_cs { · 22 unmodified lines
12131286 struct list_head debugfs_list;
12141287 u64 sequence;
12151288 u64 staged_sequence;
1289+ u64 timeout_jiffies;
12161290 enum hl_cs_type type;
12171291 u8 submitted;
12181292 u8 completed;
@@ -1329,6 +1403,23 @@ struct hl_vm_hash_node { · 110 unmodified lines
13291403 void *ptr;
13301404 };
13311405
1406+/**
1407+ * struct hl_vm_hw_block_list_node - list element from user virtual address to
1408+ * HW block id.
1409+ * @node: node to hang on the list in context object.
1410+ * @ctx: the context this node belongs to.
1411+ * @vaddr: virtual address of the HW block.
1412+ * @size: size of the block.
1413+ * @id: HW block id (handle).
1414+ */
1415+struct hl_vm_hw_block_list_node {
1416+ struct list_head node;
1417+ struct hl_ctx *ctx;
1418+ unsigned long vaddr;
1419+ u32 size;
1420+ u32 id;
1421+};
1422+
13321423 /**
13331424 * struct hl_vm_phys_pg_pack - physical page pack.
13341425 * @vm_type: describes the type of the virtual area descriptor.
@@ -1490,12 +1581,13 @@ struct hl_debugfs_entry { · 155 unmodified lines
14901581 * @userptr_spinlock: protects userptr_list.
14911582 * @ctx_mem_hash_list: list of available contexts with MMU mappings.
14921583 * @ctx_mem_hash_spinlock: protects cb_list.
1584+ * @blob_desc: descriptor of blob
14931585 * @addr: next address to read/write from/to in read/write32.
14941586 * @mmu_addr: next virtual address to translate to physical address in mmu_show.
14951587 * @mmu_asid: ASID to use while translating in mmu_show.
14961588 * @i2c_bus: generic u8 debugfs file for bus value to use in i2c_data_read.
1497− * @i2c_bus: generic u8 debugfs file for address value to use in i2c_data_read.
1498− * @i2c_bus: generic u8 debugfs file for register value to use in i2c_data_read.
1589+ * @i2c_addr: generic u8 debugfs file for address value to use in i2c_data_read.
1590+ * @i2c_reg: generic u8 debugfs file for register value to use in i2c_data_read.
14991591 */
15001592 struct hl_dbg_device_entry {
15011593 struct dentry *root;
@@ -1513,6 +1605,7 @@ struct hl_dbg_device_entry { · 11 unmodified lines
15131605 spinlock_t userptr_spinlock;
15141606 struct list_head ctx_mem_hash_list;
15151607 spinlock_t ctx_mem_hash_spinlock;
1608+ struct debugfs_blob_wrapper blob_desc;
15161609 u64 addr;
15171610 u64 mmu_addr;
15181611 u32 mmu_asid;
@@ -1683,16 +1776,6 @@ struct hl_device_reset_work { · 164 unmodified lines
16831776 struct hl_device *hdev;
16841777 };
16851778
1686−/**
1687− * struct hl_device_idle_busy_ts - used for calculating device utilization rate.
1688− * @idle_to_busy_ts: timestamp where device changed from idle to busy.
1689− * @busy_to_idle_ts: timestamp where device changed from busy to idle.
1690− */
1691−struct hl_device_idle_busy_ts {
1692− ktime_t idle_to_busy_ts;
1693− ktime_t busy_to_idle_ts;
1694−};
1695−
16961779 /**
16971780 * struct hr_mmu_hop_addrs - used for holding per-device host-resident mmu hop
16981781 * information.
@@ -1821,9 +1904,16 @@ struct hl_mmu_funcs { · 122 unmodified lines
18211904 * @asic_name: ASIC specific name.
18221905 * @asic_type: ASIC specific type.
18231906 * @completion_queue: array of hl_cq.
1907+ * @user_interrupt: array of hl_user_interrupt. upon the corresponding user
1908+ * interrupt, driver will monitor the list of fences
1909+ * registered to this interrupt.
1910+ * @common_user_interrupt: common user interrupt for all user interrupts.
1911+ * upon any user interrupt, driver will monitor the
1912+ * list of fences registered to this common structure.
18241913 * @cq_wq: work queues of completion queues for executing work in process
18251914 * context.
18261915 * @eq_wq: work queue of event queue for executing work in process context.
1916+ * @sob_reset_wq: work queue for sob reset executions.
18271917 * @kernel_ctx: Kernel driver context structure.
18281918 * @kernel_queues: array of hl_hw_queue.
18291919 * @cs_mirror_list: CS mirror list for TDR.
@@ -1857,11 +1947,11 @@ struct hl_mmu_funcs { · 27 unmodified lines
18571947 * when a user opens the device
18581948 * @fpriv_list_lock: protects the fpriv_list
18591949 * @compute_ctx: current compute context executing.
1860− * @idle_busy_ts_arr: array to hold time stamps of transitions from idle to busy
1861− * and vice-versa
18621950 * @aggregated_cs_counters: aggregated cs counters among all contexts
18631951 * @mmu_priv: device-specific MMU data.
18641952 * @mmu_func: device-related MMU functions.
1953+ * @legacy_pll_map: map holding map between dynamic (common) PLL indexes and
1954+ * static (asic specific) PLL indexes.
18651955 * @dram_used_mem: current DRAM memory consumption.
18661956 * @timeout_jiffies: device CS timeout value.
18671957 * @max_power: the max power of the device, as configured by the sysadmin. This
@@ -1874,13 +1964,10 @@ struct hl_mmu_funcs { · 6 unmodified lines
18741964 * @curr_pll_profile: current PLL profile.
18751965 * @card_type: Various ASICs have several card types. This indicates the card
18761966 * type of the current device.
1877− * @cs_active_cnt: number of active command submissions on this device (active
1878− * means already in H/W queues)
18791967 * @major: habanalabs kernel driver major.
18801968 * @high_pll: high PLL profile frequency.
18811969 * @soft_reset_cnt: number of soft reset since the driver was loaded.
18821970 * @hard_reset_cnt: number of hard reset since the driver was loaded.
1883− * @idle_busy_ts_idx: index of current entry in idle_busy_ts_arr
18841971 * @clk_throttling_reason: bitmask represents the current clk throttling reasons
18851972 * @id: device minor.
18861973 * @id_control: minor of the control device
@@ -1937,8 +2024,11 @@ struct hl_device { · 50 unmodified lines
19372024 char status[HL_DEV_STS_MAX][HL_STR_MAX];
19382025 enum hl_asic_type asic_type;
19392026 struct hl_cq *completion_queue;
2027+ struct hl_user_interrupt *user_interrupt;
2028+ struct hl_user_interrupt common_user_interrupt;
19402029 struct workqueue_struct **cq_wq;
19412030 struct workqueue_struct *eq_wq;
2031+ struct workqueue_struct *sob_reset_wq;
19422032 struct hl_ctx *kernel_ctx;
19432033 struct hl_hw_queue *kernel_queues;
19442034 struct list_head cs_mirror_list;
@@ -1976,13 +2066,13 @@ struct hl_device { · 31 unmodified lines
19762066
19772067 struct hl_ctx *compute_ctx;
19782068
1979− struct hl_device_idle_busy_ts *idle_busy_ts_arr;
1980−
19812069 struct hl_cs_counters_atomic aggregated_cs_counters;
19822070
19832071 struct hl_mmu_priv mmu_priv;
19842072 struct hl_mmu_funcs mmu_func[MMU_NUM_PGT_LOCATIONS];
19852073
2074+ enum pll_index *legacy_pll_map;
2075+
19862076 atomic64_t dram_used_mem;
19872077 u64 timeout_jiffies;
19882078 u64 max_power;
@@ -1990,12 +2080,10 @@ struct hl_device { · 1 unmodified line
19902080 atomic_t in_reset;
19912081 enum hl_pll_frequency curr_pll_profile;
19922082 enum cpucp_card_types card_type;
1993− int cs_active_cnt;
19942083 u32 major;
19952084 u32 high_pll;
19962085 u32 soft_reset_cnt;
19972086 u32 hard_reset_cnt;
1998− u32 idle_busy_ts_idx;
19992087 u32 clk_throttling_reason;
20002088 u16 id;
20012089 u16 id_control;
@@ -2029,10 +2117,9 @@ struct hl_device { · 27 unmodified lines
20292117
20302118 /* Parameters for bring-up */
20312119 u64 nic_ports_mask;
2032− u64 fw_loading;
2120+ u64 fw_components;
20332121 u8 mmu_enable;
20342122 u8 mmu_huge_page_opt;
2035− u8 cpu_enable;
20362123 u8 reset_pcilink;
20372124 u8 cpu_queues_enable;
20382125 u8 pldm;
@@ -2043,6 +2130,7 @@ struct hl_device { · 4 unmodified lines
20432130 u8 bmc_enable;
20442131 u8 rl_enable;
20452132 u8 reset_on_preboot_fail;
2133+ u8 reset_upon_device_release;
20462134 };
20472135
20482136
@@ -2157,6 +2245,8 @@ void hl_cq_reset(struct hl_device *hdev, struct hl_cq *q); · 108 unmodified lines
21572245 void hl_eq_reset(struct hl_device *hdev, struct hl_eq *q);
21582246 irqreturn_t hl_irq_handler_cq(int irq, void *arg);
21592247 irqreturn_t hl_irq_handler_eq(int irq, void *arg);
2248+irqreturn_t hl_irq_handler_user_cq(int irq, void *arg);
2249+irqreturn_t hl_irq_handler_default(int irq, void *arg);
21602250 u32 hl_cq_inc_ptr(u32 ptr);
21612251
21622252 int hl_asid_init(struct hl_device *hdev);
@@ -2178,12 +2268,11 @@ int hl_device_init(struct hl_device *hdev, struct class *hclass); · 15 unmodified lines
21782268 void hl_device_fini(struct hl_device *hdev);
21792269 int hl_device_suspend(struct hl_device *hdev);
21802270 int hl_device_resume(struct hl_device *hdev);
2181−int hl_device_reset(struct hl_device *hdev, bool hard_reset,
2182− bool from_hard_reset_thread);
2271+int hl_device_reset(struct hl_device *hdev, u32 flags);
21832272 void hl_hpriv_get(struct hl_fpriv *hpriv);
2184−void hl_hpriv_put(struct hl_fpriv *hpriv);
2273+int hl_hpriv_put(struct hl_fpriv *hpriv);
21852274 int hl_device_set_frequency(struct hl_device *hdev, enum hl_pll_frequency freq);
2186−uint32_t hl_device_utilization(struct hl_device *hdev, uint32_t period_ms);
2275+int hl_device_utilization(struct hl_device *hdev, u32 *utilization);
21872276
21882277 int hl_build_hwmon_channel_info(struct hl_device *hdev,
21892278 struct cpucp_sensor *sensors_arr);
@@ -2235,6 +2324,9 @@ void hl_vm_ctx_fini(struct hl_ctx *ctx); · 45 unmodified lines
22352324 int hl_vm_init(struct hl_device *hdev);
22362325 void hl_vm_fini(struct hl_device *hdev);
22372326
2327+void hl_hw_block_mem_init(struct hl_ctx *ctx);
2328+void hl_hw_block_mem_fini(struct hl_ctx *ctx);
2329+
22382330 u64 hl_reserve_va_block(struct hl_device *hdev, struct hl_ctx *ctx,
22392331 enum hl_va_range_type type, u32 size, u32 alignment);
22402332 int hl_unreserve_va_block(struct hl_device *hdev, struct hl_ctx *ctx,
@@ -2287,13 +2379,19 @@ int hl_fw_send_heartbeat(struct hl_device *hdev); · 46 unmodified lines
22872379 int hl_fw_cpucp_info_get(struct hl_device *hdev,
22882380 u32 cpu_security_boot_status_reg,
22892381 u32 boot_err0_reg);
2382+int hl_fw_cpucp_handshake(struct hl_device *hdev,
2383+ u32 cpu_security_boot_status_reg,
2384+ u32 boot_err0_reg);
22902385 int hl_fw_get_eeprom_data(struct hl_device *hdev, void *data, size_t max_size);
22912386 int hl_fw_cpucp_pci_counters_get(struct hl_device *hdev,
22922387 struct hl_info_pci_counters *counters);
22932388 int hl_fw_cpucp_total_energy_get(struct hl_device *hdev,
22942389 u64 *total_energy);
2295−int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, u16 pll_index,
2390+int get_used_pll_index(struct hl_device *hdev, enum pll_index input_pll_index,
2391+ enum pll_index *pll_index);
2392+int hl_fw_cpucp_pll_info_get(struct hl_device *hdev, enum pll_index pll_index,
22962393 u16 *pll_freq_arr);
2394+int hl_fw_cpucp_power_get(struct hl_device *hdev, u64 *power);
22972395 int hl_fw_init_cpu(struct hl_device *hdev, u32 cpu_boot_status_reg,
22982396 u32 msg_to_cpu_reg, u32 cpu_msg_status_reg,
22992397 u32 cpu_security_boot_status_reg, u32 boot_err0_reg,
@@ -2304,6 +2402,7 @@ int hl_fw_read_preboot_status(struct hl_device *hdev, u32 cpu_boot_status_reg, · 4 unmodified lines
23042402
23052403 int hl_pci_bars_map(struct hl_device *hdev, const char * const name[3],
23062404 bool is_wc[3]);
2405+int hl_pci_elbi_read(struct hl_device *hdev, u64 addr, u32 *data);
23072406 int hl_pci_iatu_write(struct hl_device *hdev, u32 addr, u32 data);
23082407 int hl_pci_set_inbound_region(struct hl_device *hdev, u8 region,
23092408 struct hl_inbound_pci_region *pci_region);
@@ -2312,8 +2411,10 @@ int hl_pci_set_outbound_region(struct hl_device *hdev, · 2 unmodified lines
23122411 int hl_pci_init(struct hl_device *hdev);
23132412 void hl_pci_fini(struct hl_device *hdev);
23142413
2315−long hl_get_frequency(struct hl_device *hdev, u32 pll_index, bool curr);
2316−void hl_set_frequency(struct hl_device *hdev, u32 pll_index, u64 freq);
2414+long hl_get_frequency(struct hl_device *hdev, enum pll_index pll_index,
2415+ bool curr);
2416+void hl_set_frequency(struct hl_device *hdev, enum pll_index pll_index,
2417+ u64 freq);
23172418 int hl_get_temperature(struct hl_device *hdev,
23182419 int sensor_index, u32 attr, long *value);
23192420 int hl_set_temperature(struct hl_device *hdev,
@@ -2334,6 +2435,7 @@ int hl_set_voltage(struct hl_device *hdev, · 14 unmodified lines
23342435 int sensor_index, u32 attr, long value);
23352436 int hl_set_current(struct hl_device *hdev,
23362437 int sensor_index, u32 attr, long value);
2438+void hl_release_pending_user_interrupts(struct hl_device *hdev);
23372439
23382440 #ifdef CONFIG_DEBUG_FS
23392441
@@ -2434,7 +2536,7 @@ long hl_ioctl(struct file *filep, unsigned int cmd, unsigned long arg); · 94 unmodified lines
24342536 long hl_ioctl_control(struct file *filep, unsigned int cmd, unsigned long arg);
24352537 int hl_cb_ioctl(struct hl_fpriv *hpriv, void *data);
24362538 int hl_cs_ioctl(struct hl_fpriv *hpriv, void *data);
2437−int hl_cs_wait_ioctl(struct hl_fpriv *hpriv, void *data);
2539+int hl_wait_ioctl(struct hl_fpriv *hpriv, void *data);
24382540 int hl_mem_ioctl(struct hl_fpriv *hpriv, void *data);
24392541
24402542 #endif /* HABANALABSP_H_ */
drivers/misc/habanalabs/common/habanalabs_drv.c+24−4
@@ -27,13 +27,13 @@ static struct class *hl_class; · 26 unmodified lines
2727 static DEFINE_IDR(hl_devs_idr);
2828 static DEFINE_MUTEX(hl_devs_idr_lock);
2929
30−static int timeout_locked = 5;
30+static int timeout_locked = 30;
3131 static int reset_on_lockup = 1;
3232 static int memory_scrub = 1;
3333
3434 module_param(timeout_locked, int, 0444);
3535 MODULE_PARM_DESC(timeout_locked,
36− "Device lockup timeout in seconds (0 = disabled, default 5s)");
36+ "Device lockup timeout in seconds (0 = disabled, default 30s)");
3737
3838 module_param(reset_on_lockup, int, 0444);
3939 MODULE_PARM_DESC(reset_on_lockup,
@@ -47,10 +47,12 @@ MODULE_PARM_DESC(memory_scrub, · 7 unmodified lines
4747
4848 #define PCI_IDS_GOYA 0x0001
4949 #define PCI_IDS_GAUDI 0x1000
50+#define PCI_IDS_GAUDI_SEC 0x1010
5051
5152 static const struct pci_device_id ids[] = {
5253 { PCI_DEVICE(PCI_VENDOR_ID_HABANALABS, PCI_IDS_GOYA), },
5354 { PCI_DEVICE(PCI_VENDOR_ID_HABANALABS, PCI_IDS_GAUDI), },
55+ { PCI_DEVICE(PCI_VENDOR_ID_HABANALABS, PCI_IDS_GAUDI_SEC), },
5456 { 0, }
5557 };
5658 MODULE_DEVICE_TABLE(pci, ids);
@@ -74,6 +76,9 @@ static enum hl_asic_type get_asic_type(u16 device) · 17 unmodified lines
7476 case PCI_IDS_GAUDI:
7577 asic_type = ASIC_GAUDI;
7678 break;
79+ case PCI_IDS_GAUDI_SEC:
80+ asic_type = ASIC_GAUDI_SEC;
81+ break;
7782 default:
7883 asic_type = ASIC_INVALID;
7984 break;
@@ -82,6 +87,16 @@ static enum hl_asic_type get_asic_type(u16 device) · 2 unmodified lines
8287 return asic_type;
8388 }
8489
90+static bool is_asic_secured(enum hl_asic_type asic_type)
91+{
92+ switch (asic_type) {
93+ case ASIC_GAUDI_SEC:
94+ return true;
95+ default:
96+ return false;
97+ }
98+}
99+
85100 /*
86101 * hl_device_open - open function for habanalabs device
87102 *
@@ -234,8 +249,7 @@ out_err: · 146 unmodified lines
234249
235250 static void set_driver_behavior_per_device(struct hl_device *hdev)
236251 {
237− hdev->cpu_enable = 1;
238− hdev->fw_loading = FW_TYPE_ALL_TYPES;
252+ hdev->fw_components = FW_TYPE_ALL_TYPES;
239253 hdev->cpu_queues_enable = 1;
240254 hdev->heartbeat = 1;
241255 hdev->mmu_enable = 1;
@@ -288,6 +302,12 @@ int create_hdev(struct hl_device **dev, struct pci_dev *pdev, · 46 unmodified lines
288302 hdev->asic_type = asic_type;
289303 }
290304
305+ if (pdev)
306+ hdev->asic_prop.fw_security_disabled =
307+ !is_asic_secured(pdev->device);
308+ else
309+ hdev->asic_prop.fw_security_disabled = true;
310+
291311 /* Assign status description string */
292312 strncpy(hdev->status[HL_DEVICE_STATUS_MALFUNCTION],
293313 "disabled", HL_STR_MAX);
drivers/misc/habanalabs/common/habanalabs_ioctl.c+26−9
@@ -226,19 +226,14 @@ static int device_utilization(struct hl_device *hdev, struct hl_info_args *args) · 225 unmodified lines
226226 struct hl_info_device_utilization device_util = {0};
227227 u32 max_size = args->return_size;
228228 void __user *out = (void __user *) (uintptr_t) args->return_pointer;
229+ int rc;
229230
230231 if ((!max_size) || (!out))
231232 return -EINVAL;
232233
233− if ((args->period_ms < 100) || (args->period_ms > 1000) ||
234− (args->period_ms % 100)) {
235− dev_err(hdev->dev,
236− "period %u must be between 100 - 1000 and must be divisible by 100\n",
237− args->period_ms);
234+ rc = hl_device_utilization(hdev, &device_util.utilization);
235+ if (rc)
238236 return -EINVAL;
239− }
240−
241− device_util.utilization = hl_device_utilization(hdev, args->period_ms);
242237
243238 return copy_to_user(out, &device_util,
244239 min((size_t) max_size, sizeof(device_util))) ? -EFAULT : 0;
@@ -446,6 +441,25 @@ static int pll_frequency_info(struct hl_fpriv *hpriv, struct hl_info_args *args) · 201 unmodified lines
446441 min((size_t) max_size, sizeof(freq_info))) ? -EFAULT : 0;
447442 }
448443
444+static int power_info(struct hl_fpriv *hpriv, struct hl_info_args *args)
445+{
446+ struct hl_device *hdev = hpriv->hdev;
447+ u32 max_size = args->return_size;
448+ struct hl_power_info power_info = {0};
449+ void __user *out = (void __user *) (uintptr_t) args->return_pointer;
450+ int rc;
451+
452+ if ((!max_size) || (!out))
453+ return -EINVAL;
454+
455+ rc = hl_fw_cpucp_power_get(hdev, &power_info.power);
456+ if (rc)
457+ return rc;
458+
459+ return copy_to_user(out, &power_info,
460+ min((size_t) max_size, sizeof(power_info))) ? -EFAULT : 0;
461+}
462+
449463 static int _hl_info_ioctl(struct hl_fpriv *hpriv, void *data,
450464 struct device *dev)
451465 {
@@ -526,6 +540,9 @@ static int _hl_info_ioctl(struct hl_fpriv *hpriv, void *data, · 74 unmodified lines
526540 case HL_INFO_PLL_FREQUENCY:
527541 return pll_frequency_info(hpriv, args);
528542
543+ case HL_INFO_POWER:
544+ return power_info(hpriv, args);
545+
529546 default:
530547 dev_err(dev, "Invalid request %d\n", args->op);
531548 rc = -ENOTTY;
@@ -596,7 +613,7 @@ static const struct hl_ioctl_desc hl_ioctls[] = { · 64 unmodified lines
596613 HL_IOCTL_DEF(HL_IOCTL_INFO, hl_info_ioctl),
597614 HL_IOCTL_DEF(HL_IOCTL_CB, hl_cb_ioctl),
598615 HL_IOCTL_DEF(HL_IOCTL_CS, hl_cs_ioctl),
599− HL_IOCTL_DEF(HL_IOCTL_WAIT_CS, hl_cs_wait_ioctl),
616+ HL_IOCTL_DEF(HL_IOCTL_WAIT_CS, hl_wait_ioctl),
600617 HL_IOCTL_DEF(HL_IOCTL_MEMORY, hl_mem_ioctl),
601618 HL_IOCTL_DEF(HL_IOCTL_DEBUG, hl_debug_ioctl)
602619 };
drivers/misc/habanalabs/common/hw_queue.c+1−9
@@ -629,20 +629,12 @@ int hl_hw_queue_schedule_cs(struct hl_cs *cs) · 628 unmodified lines
629629 if ((hdev->timeout_jiffies != MAX_SCHEDULE_TIMEOUT) &&
630630 first_entry && cs_needs_timeout(cs)) {
631631 cs->tdr_active = true;
632− schedule_delayed_work(&cs->work_tdr, hdev->timeout_jiffies);
632+ schedule_delayed_work(&cs->work_tdr, cs->timeout_jiffies);
633633
634634 }
635635
636636 spin_unlock(&hdev->cs_mirror_lock);
637637
638− if (!hdev->cs_active_cnt++) {
639− struct hl_device_idle_busy_ts *ts;
640−
641− ts = &hdev->idle_busy_ts_arr[hdev->idle_busy_ts_idx];
642− ts->busy_to_idle_ts = ktime_set(0, 0);
643− ts->idle_to_busy_ts = ktime_get();
644− }
645−
646638 list_for_each_entry_safe(job, tmp, &cs->job_list, cs_node)
647639 switch (job->queue_type) {
648640 case QUEUE_TYPE_EXT:
drivers/misc/habanalabs/common/irq.c+56−0
@@ -137,6 +137,62 @@ irqreturn_t hl_irq_handler_cq(int irq, void *arg) · 136 unmodified lines
137137 return IRQ_HANDLED;
138138 }
139139
140+static void handle_user_cq(struct hl_device *hdev,
141+ struct hl_user_interrupt *user_cq)
142+{
143+ struct hl_user_pending_interrupt *pend;
144+
145+ spin_lock(&user_cq->wait_list_lock);
146+ list_for_each_entry(pend, &user_cq->wait_list_head, wait_list_node)
147+ complete_all(&pend->fence.completion);
148+ spin_unlock(&user_cq->wait_list_lock);
149+}
150+
151+/**
152+ * hl_irq_handler_user_cq - irq handler for user completion queues
153+ *
154+ * @irq: irq number
155+ * @arg: pointer to user interrupt structure
156+ *
157+ */
158+irqreturn_t hl_irq_handler_user_cq(int irq, void *arg)
159+{
160+ struct hl_user_interrupt *user_cq = arg;
161+ struct hl_device *hdev = user_cq->hdev;
162+
163+ dev_dbg(hdev->dev,
164+ "got user completion interrupt id %u",
165+ user_cq->interrupt_id);
166+
167+ /* Handle user cq interrupts registered on all interrupts */
168+ handle_user_cq(hdev, &hdev->common_user_interrupt);
169+
170+ /* Handle user cq interrupts registered on this specific interrupt */
171+ handle_user_cq(hdev, user_cq);
172+
173+ return IRQ_HANDLED;
174+}
175+
176+/**
177+ * hl_irq_handler_default - default irq handler
178+ *
179+ * @irq: irq number
180+ * @arg: pointer to user interrupt structure
181+ *
182+ */
183+irqreturn_t hl_irq_handler_default(int irq, void *arg)
184+{
185+ struct hl_user_interrupt *user_interrupt = arg;
186+ struct hl_device *hdev = user_interrupt->hdev;
187+ u32 interrupt_id = user_interrupt->interrupt_id;
188+
189+ dev_err(hdev->dev,
190+ "got invalid user interrupt %u",
191+ interrupt_id);
192+
193+ return IRQ_HANDLED;
194+}
195+
140196 /**
141197 * hl_irq_handler_eq - irq handler for event queue
142198 *
drivers/misc/habanalabs/common/memory.c+129−53
@@ -81,16 +81,6 @@ static int alloc_device_memory(struct hl_ctx *ctx, struct hl_mem_in *args, · 80 unmodified lines
8181 num_pgs, total_size);
8282 return -ENOMEM;
8383 }
84−
85− if (hdev->memory_scrub) {
86− rc = hdev->asic_funcs->scrub_device_mem(hdev, paddr,
87− total_size);
88− if (rc) {
89− dev_err(hdev->dev,
90− "Failed to scrub contiguous device memory\n");
91− goto pages_pack_err;
92− }
93− }
9484 }
9585
9686 phys_pg_pack = kzalloc(sizeof(*phys_pg_pack), GFP_KERNEL);
@@ -128,24 +118,13 @@ static int alloc_device_memory(struct hl_ctx *ctx, struct hl_mem_in *args, · 31 unmodified lines
128118 goto page_err;
129119 }
130120
131− if (hdev->memory_scrub) {
132− rc = hdev->asic_funcs->scrub_device_mem(hdev,
133− phys_pg_pack->pages[i],
134− page_size);
135− if (rc) {
136− dev_err(hdev->dev,
137− "Failed to scrub device memory\n");
138− goto page_err;
139− }
140− }
141−
142121 num_curr_pgs++;
143122 }
144123 }
145124
146125 spin_lock(&vm->idr_lock);
147126 handle = idr_alloc(&vm->phys_pg_pack_handles, phys_pg_pack, 1, 0,
148− GFP_ATOMIC);
127+ GFP_KERNEL);
149128 spin_unlock(&vm->idr_lock);
150129
151130 if (handle < 0) {
@@ -280,37 +259,67 @@ static void dram_pg_pool_do_release(struct kref *ref) · 128 unmodified lines
280259 * @phys_pg_pack: physical page pack to free.
281260 *
282261 * This function does the following:
283− * - For DRAM memory only, iterate over the pack and free each physical block
284− * structure by returning it to the general pool.
262+ * - For DRAM memory only
263+ * - iterate over the pack, scrub and free each physical block structure by
264+ * returning it to the general pool.
265+ * In case of error during scrubbing, initiate hard reset.
266+ * Once hard reset is triggered, scrubbing is bypassed while freeing the
267+ * memory continues.
285268 * - Free the hl_vm_phys_pg_pack structure.
286269 */
287−static void free_phys_pg_pack(struct hl_device *hdev,
270+static int free_phys_pg_pack(struct hl_device *hdev,
288271 struct hl_vm_phys_pg_pack *phys_pg_pack)
289272 {
290273 struct hl_vm *vm = &hdev->vm;
291274 u64 i;
275+ int rc = 0;
292276
293− if (!phys_pg_pack->created_from_userptr) {
294− if (phys_pg_pack->contiguous) {
295− gen_pool_free(vm->dram_pg_pool, phys_pg_pack->pages[0],
277+ if (phys_pg_pack->created_from_userptr)
278+ goto end;
279+
280+ if (phys_pg_pack->contiguous) {
281+ if (hdev->memory_scrub && !hdev->disabled) {
282+ rc = hdev->asic_funcs->scrub_device_mem(hdev,
283+ phys_pg_pack->pages[0],
296284 phys_pg_pack->total_size);
285+ if (rc)
286+ dev_err(hdev->dev,
287+ "Failed to scrub contiguous device memory\n");
288+ }
297289
298− for (i = 0; i < phys_pg_pack->npages ; i++)
299− kref_put(&vm->dram_pg_pool_refcount,
300− dram_pg_pool_do_release);
301− } else {
302− for (i = 0 ; i < phys_pg_pack->npages ; i++) {
303− gen_pool_free(vm->dram_pg_pool,
290+ gen_pool_free(vm->dram_pg_pool, phys_pg_pack->pages[0],
291+ phys_pg_pack->total_size);
292+
293+ for (i = 0; i < phys_pg_pack->npages ; i++)
294+ kref_put(&vm->dram_pg_pool_refcount,
295+ dram_pg_pool_do_release);
296+ } else {
297+ for (i = 0 ; i < phys_pg_pack->npages ; i++) {
298+ if (hdev->memory_scrub && !hdev->disabled && rc == 0) {
299+ rc = hdev->asic_funcs->scrub_device_mem(
300+ hdev,
304301 phys_pg_pack->pages[i],
305302 phys_pg_pack->page_size);
306− kref_put(&vm->dram_pg_pool_refcount,
307− dram_pg_pool_do_release);
303+ if (rc)
304+ dev_err(hdev->dev,
305+ "Failed to scrub device memory\n");
308306 }
307+ gen_pool_free(vm->dram_pg_pool,
308+ phys_pg_pack->pages[i],
309+ phys_pg_pack->page_size);
310+ kref_put(&vm->dram_pg_pool_refcount,
311+ dram_pg_pool_do_release);
309312 }
310313 }
311314
315+ if (rc && !hdev->disabled)
316+ hl_device_reset(hdev, HL_RESET_HARD);
317+
318+end:
312319 kvfree(phys_pg_pack->pages);
313320 kfree(phys_pg_pack);
321+
322+ return rc;
314323 }
315324
316325 /**
@@ -349,7 +358,7 @@ static int free_device_memory(struct hl_ctx *ctx, struct hl_mem_in *args) · 32 unmodified lines
349358 atomic64_sub(phys_pg_pack->total_size, &ctx->dram_phys_mem);
350359 atomic64_sub(phys_pg_pack->total_size, &hdev->dram_used_mem);
351360
352− free_phys_pg_pack(hdev, phys_pg_pack);
361+ return free_phys_pg_pack(hdev, phys_pg_pack);
353362 } else {
354363 spin_unlock(&vm->idr_lock);
355364 dev_err(hdev->dev,
@@ -857,6 +866,7 @@ static int map_phys_pg_pack(struct hl_ctx *ctx, u64 vaddr, · 501 unmodified lines
857866 u64 next_vaddr = vaddr, paddr, mapped_pg_cnt = 0, i;
858867 u32 page_size = phys_pg_pack->page_size;
859868 int rc = 0;
869+ bool is_host_addr;
860870
861871 for (i = 0 ; i < phys_pg_pack->npages ; i++) {
862872 paddr = phys_pg_pack->pages[i];
@@ -878,6 +888,8 @@ static int map_phys_pg_pack(struct hl_ctx *ctx, u64 vaddr, · 15 unmodified lines
878888 return 0;
879889
880890 err:
891+ is_host_addr = !hl_is_dram_va(hdev, vaddr);
892+
881893 next_vaddr = vaddr;
882894 for (i = 0 ; i < mapped_pg_cnt ; i++) {
883895 if (hl_mmu_unmap_page(ctx, next_vaddr, page_size,
@@ -888,6 +900,17 @@ err: · 4 unmodified lines
888900 phys_pg_pack->pages[i], page_size);
889901
890902 next_vaddr += page_size;
903+
904+ /*
905+ * unmapping on Palladium can be really long, so avoid a CPU
906+ * soft lockup bug by sleeping a little between unmapping pages
907+ *
908+ * In addition, on host num of pages could be huge,
909+ * because page size could be 4KB, so when unmapping host
910+ * pages sleep every 32K pages to avoid soft lockup
911+ */
912+ if (hdev->pldm || (is_host_addr && (i & 0x7FFF) == 0))
913+ usleep_range(50, 200);
891914 }
892915
893916 return rc;
@@ -921,9 +944,9 @@ static void unmap_phys_pg_pack(struct hl_ctx *ctx, u64 vaddr, · 27 unmodified lines
921944 * unmapping on Palladium can be really long, so avoid a CPU
922945 * soft lockup bug by sleeping a little between unmapping pages
923946 *
924− * In addition, when unmapping host memory we pass through
925− * the Linux kernel to unpin the pages and that takes a long
926− * time. Therefore, sleep every 32K pages to avoid soft lockup
947+ * In addition, on host num of pages could be huge,
948+ * because page size could be 4KB, so when unmapping host
949+ * pages sleep every 32K pages to avoid soft lockup
927950 */
928951 if (hdev->pldm || (is_host_addr && (i & 0x7FFF) == 0))
929952 usleep_range(50, 200);
@@ -1117,9 +1140,9 @@ static int map_device_va(struct hl_ctx *ctx, struct hl_mem_in *args, · 187 unmodified lines
11171140 *device_addr = ret_vaddr;
11181141
11191142 if (is_userptr)
1120− free_phys_pg_pack(hdev, phys_pg_pack);
1143+ rc = free_phys_pg_pack(hdev, phys_pg_pack);
11211144
1122− return 0;
1145+ return rc;
11231146
11241147 map_err:
11251148 if (add_va_block(hdev, va_range, ret_vaddr,
@@ -1272,7 +1295,7 @@ static int unmap_device_va(struct hl_ctx *ctx, struct hl_mem_in *args, · 146 unmodified lines
12721295 kfree(hnode);
12731296
12741297 if (is_userptr) {
1275− free_phys_pg_pack(hdev, phys_pg_pack);
1298+ rc = free_phys_pg_pack(hdev, phys_pg_pack);
12761299 dma_unmap_host_va(hdev, userptr);
12771300 }
12781301
@@ -1305,9 +1328,15 @@ static int map_block(struct hl_device *hdev, u64 address, u64 *handle, · 26 unmodified lines
13051328
13061329 static void hw_block_vm_close(struct vm_area_struct *vma)
13071330 {
1308− struct hl_ctx *ctx = (struct hl_ctx *) vma->vm_private_data;
1331+ struct hl_vm_hw_block_list_node *lnode =
1332+ (struct hl_vm_hw_block_list_node *) vma->vm_private_data;
1333+ struct hl_ctx *ctx = lnode->ctx;
13091334
1335+ mutex_lock(&ctx->hw_block_list_lock);
1336+ list_del(&lnode->node);
1337+ mutex_unlock(&ctx->hw_block_list_lock);
13101338 hl_ctx_put(ctx);
1339+ kfree(lnode);
13111340 vma->vm_private_data = NULL;
13121341 }
13131342
@@ -1325,7 +1354,9 @@ static const struct vm_operations_struct hw_block_vm_ops = { · 11 unmodified lines
13251354 */
13261355 int hl_hw_block_mmap(struct hl_fpriv *hpriv, struct vm_area_struct *vma)
13271356 {
1357+ struct hl_vm_hw_block_list_node *lnode;
13281358 struct hl_device *hdev = hpriv->hdev;
1359+ struct hl_ctx *ctx = hpriv->ctx;
13291360 u32 block_id, block_size;
13301361 int rc;
13311362
@@ -1351,17 +1382,31 @@ int hl_hw_block_mmap(struct hl_fpriv *hpriv, struct vm_area_struct *vma) · 19 unmodified lines
13511382 return -EINVAL;
13521383 }
13531384
1385+ lnode = kzalloc(sizeof(*lnode), GFP_KERNEL);
1386+ if (!lnode)
1387+ return -ENOMEM;
1388+
13541389 vma->vm_ops = &hw_block_vm_ops;
1355− vma->vm_private_data = hpriv->ctx;
1390+ vma->vm_private_data = lnode;
13561391
1357− hl_ctx_get(hdev, hpriv->ctx);
1392+ hl_ctx_get(hdev, ctx);
13581393
13591394 rc = hdev->asic_funcs->hw_block_mmap(hdev, vma, block_id, block_size);
13601395 if (rc) {
1361− hl_ctx_put(hpriv->ctx);
1396+ hl_ctx_put(ctx);
1397+ kfree(lnode);
13621398 return rc;
13631399 }
13641400
1401+ lnode->ctx = ctx;
1402+ lnode->vaddr = vma->vm_start;
1403+ lnode->size = block_size;
1404+ lnode->id = block_id;
1405+
1406+ mutex_lock(&ctx->hw_block_list_lock);
1407+ list_add_tail(&lnode->node, &ctx->hw_block_mem_list);
1408+ mutex_unlock(&ctx->hw_block_list_lock);
1409+
13651410 vma->vm_pgoff = block_id;
13661411
13671412 return 0;
@@ -1574,7 +1619,7 @@ static int get_user_memory(struct hl_device *hdev, u64 addr, u64 size, · 206 unmodified lines
15741619
15751620 rc = sg_alloc_table_from_pages(userptr->sgt,
15761621 userptr->pages,
1577− npages, offset, size, GFP_ATOMIC);
1622+ npages, offset, size, GFP_KERNEL);
15781623 if (rc < 0) {
15791624 dev_err(hdev->dev, "failed to create SG table from pages\n");
15801625 goto put_pages;
@@ -1624,11 +1669,7 @@ int hl_pin_host_memory(struct hl_device *hdev, u64 addr, u64 size, · 43 unmodified lines
16241669 return -EINVAL;
16251670 }
16261671
1627− /*
1628− * This function can be called also from data path, hence use atomic
1629− * always as it is not a big allocation.
1630− */
1631− userptr->sgt = kzalloc(sizeof(*userptr->sgt), GFP_ATOMIC);
1672+ userptr->sgt = kzalloc(sizeof(*userptr->sgt), GFP_KERNEL);
16321673 if (!userptr->sgt)
16331674 return -ENOMEM;
16341675
@@ -2122,3 +2163,38 @@ void hl_vm_fini(struct hl_device *hdev) · 487 unmodified lines
21222163
21232164 vm->init_done = false;
21242165 }
2166+
2167+/**
2168+ * hl_hw_block_mem_init() - HW block memory initialization.
2169+ * @ctx: pointer to the habanalabs context structure.
2170+ *
2171+ * This function initializes the HW block virtual mapped addresses list and
2172+ * it's lock.
2173+ */
2174+void hl_hw_block_mem_init(struct hl_ctx *ctx)
2175+{
2176+ mutex_init(&ctx->hw_block_list_lock);
2177+ INIT_LIST_HEAD(&ctx->hw_block_mem_list);
2178+}
2179+
2180+/**
2181+ * hl_hw_block_mem_fini() - HW block memory teardown.
2182+ * @ctx: pointer to the habanalabs context structure.
2183+ *
2184+ * This function clears the HW block virtual mapped addresses list and destroys
2185+ * it's lock.
2186+ */
2187+void hl_hw_block_mem_fini(struct hl_ctx *ctx)
2188+{
2189+ struct hl_vm_hw_block_list_node *lnode, *tmp;
2190+
2191+ if (!list_empty(&ctx->hw_block_mem_list))
2192+ dev_crit(ctx->hdev->dev, "HW block mem list isn't empty\n");
2193+
2194+ list_for_each_entry_safe(lnode, tmp, &ctx->hw_block_mem_list, node) {
2195+ list_del(&lnode->node);
2196+ kfree(lnode);
2197+ }
2198+
2199+ mutex_destroy(&ctx->hw_block_list_lock);
2200+}
drivers/misc/habanalabs/common/mmu/mmu.c+3−0
@@ -532,6 +532,8 @@ int hl_mmu_va_to_pa(struct hl_ctx *ctx, u64 virt_addr, u64 *phys_addr) · 531 unmodified lines
532532 struct hl_mmu_hop_info hops;
533533 int rc;
534534
535+ memset(&hops, 0, sizeof(hops));
536+
535537 rc = hl_mmu_get_tlb_info(ctx, virt_addr, &hops);
536538 if (rc)
537539 return rc;
@@ -589,6 +591,7 @@ int hl_mmu_if_set_funcs(struct hl_device *hdev) · 51 unmodified lines
589591 switch (hdev->asic_type) {
590592 case ASIC_GOYA:
591593 case ASIC_GAUDI:
594+ case ASIC_GAUDI_SEC:
592595 hl_mmu_v1_set_funcs(hdev, &hdev->mmu_func[MMU_DR_PGT]);
593596 break;
594597 default:
drivers/misc/habanalabs/common/pci/pci.c+52−0
@@ -85,6 +85,58 @@ static void hl_pci_bars_unmap(struct hl_device *hdev) · 84 unmodified lines
8585 pci_release_regions(pdev);
8686 }
8787
88+int hl_pci_elbi_read(struct hl_device *hdev, u64 addr, u32 *data)
89+{
90+ struct pci_dev *pdev = hdev->pdev;
91+ ktime_t timeout;
92+ u64 msec;
93+ u32 val;
94+
95+ if (hdev->pldm)
96+ msec = HL_PLDM_PCI_ELBI_TIMEOUT_MSEC;
97+ else
98+ msec = HL_PCI_ELBI_TIMEOUT_MSEC;
99+
100+ /* Clear previous status */
101+ pci_write_config_dword(pdev, mmPCI_CONFIG_ELBI_STS, 0);
102+
103+ pci_write_config_dword(pdev, mmPCI_CONFIG_ELBI_ADDR, (u32) addr);
104+ pci_write_config_dword(pdev, mmPCI_CONFIG_ELBI_CTRL, 0);
105+
106+ timeout = ktime_add_ms(ktime_get(), msec);
107+ for (;;) {
108+ pci_read_config_dword(pdev, mmPCI_CONFIG_ELBI_STS, &val);
109+ if (val & PCI_CONFIG_ELBI_STS_MASK)
110+ break;
111+ if (ktime_compare(ktime_get(), timeout) > 0) {
112+ pci_read_config_dword(pdev, mmPCI_CONFIG_ELBI_STS,
113+ &val);
114+ break;
115+ }
116+
117+ usleep_range(300, 500);
118+ }
119+
120+ if ((val & PCI_CONFIG_ELBI_STS_MASK) == PCI_CONFIG_ELBI_STS_DONE) {
121+ pci_read_config_dword(pdev, mmPCI_CONFIG_ELBI_DATA, data);
122+
123+ return 0;
124+ }
125+
126+ if (val & PCI_CONFIG_ELBI_STS_ERR) {
127+ dev_err(hdev->dev, "Error reading from ELBI\n");
128+ return -EIO;
129+ }
130+
131+ if (!(val & PCI_CONFIG_ELBI_STS_MASK)) {
132+ dev_err(hdev->dev, "ELBI read didn't finish in time\n");
133+ return -EIO;
134+ }
135+
136+ dev_err(hdev->dev, "ELBI read has undefined bits in status\n");
137+ return -EIO;
138+}
139+
88140 /**
89141 * hl_pci_elbi_write() - Write through the ELBI interface.
90142 * @hdev: Pointer to hl_device structure.
drivers/misc/habanalabs/common/sysfs.c+24−9
@@ -9,12 +9,18 @@ · 8 unmodified lines
99
1010 #include <linux/pci.h>
1111
12−long hl_get_frequency(struct hl_device *hdev, u32 pll_index, bool curr)
12+long hl_get_frequency(struct hl_device *hdev, enum pll_index pll_index,
13+ bool curr)
1314 {
1415 struct cpucp_packet pkt;
16+ u32 used_pll_idx;
1517 u64 result;
1618 int rc;
1719
20+ rc = get_used_pll_index(hdev, pll_index, &used_pll_idx);
21+ if (rc)
22+ return rc;
23+
1824 memset(&pkt, 0, sizeof(pkt));
1925
2026 if (curr)
@@ -23,7 +29,7 @@ long hl_get_frequency(struct hl_device *hdev, u32 pll_index, bool curr) · 2 unmodified lines
2329 else
2430 pkt.ctl = cpu_to_le32(CPUCP_PACKET_FREQUENCY_GET <<
2531 CPUCP_PKT_CTL_OPCODE_SHIFT);
26− pkt.pll_index = cpu_to_le32(pll_index);
32+ pkt.pll_index = cpu_to_le32((u32)used_pll_idx);
2733
2834 rc = hdev->asic_funcs->send_cpu_message(hdev, (u32 *) &pkt, sizeof(pkt),
2935 0, &result);
@@ -31,23 +37,29 @@ long hl_get_frequency(struct hl_device *hdev, u32 pll_index, bool curr) · 1 unmodified line
3137 if (rc) {
3238 dev_err(hdev->dev,
3339 "Failed to get frequency of PLL %d, error %d\n",
34− pll_index, rc);
40+ used_pll_idx, rc);
3541 return rc;
3642 }
3743
3844 return (long) result;
3945 }
4046
41−void hl_set_frequency(struct hl_device *hdev, u32 pll_index, u64 freq)
47+void hl_set_frequency(struct hl_device *hdev, enum pll_index pll_index,
48+ u64 freq)
4249 {
4350 struct cpucp_packet pkt;
51+ u32 used_pll_idx;
4452 int rc;
4553
54+ rc = get_used_pll_index(hdev, pll_index, &used_pll_idx);
55+ if (rc)
56+ return;
57+
4658 memset(&pkt, 0, sizeof(pkt));
4759
4860 pkt.ctl = cpu_to_le32(CPUCP_PACKET_FREQUENCY_SET <<
4961 CPUCP_PKT_CTL_OPCODE_SHIFT);
50− pkt.pll_index = cpu_to_le32(pll_index);
62+ pkt.pll_index = cpu_to_le32((u32)used_pll_idx);
5163 pkt.value = cpu_to_le64(freq);
5264
5365 rc = hdev->asic_funcs->send_cpu_message(hdev, (u32 *) &pkt, sizeof(pkt),
@@ -56,7 +68,7 @@ void hl_set_frequency(struct hl_device *hdev, u32 pll_index, u64 freq) · 2 unmodified lines
5668 if (rc)
5769 dev_err(hdev->dev,
5870 "Failed to set frequency to PLL %d, error %d\n",
59− pll_index, rc);
71+ used_pll_idx, rc);
6072 }
6173
6274 u64 hl_get_max_power(struct hl_device *hdev)
@@ -203,7 +215,7 @@ static ssize_t soft_reset_store(struct device *dev, · 140 unmodified lines
203215
204216 dev_warn(hdev->dev, "Soft-Reset requested through sysfs\n");
205217
206− hl_device_reset(hdev, false, false);
218+ hl_device_reset(hdev, 0);
207219
208220 out:
209221 return count;
@@ -226,7 +238,7 @@ static ssize_t hard_reset_store(struct device *dev, · 16 unmodified lines
226238
227239 dev_warn(hdev->dev, "Hard-Reset requested through sysfs\n");
228240
229− hl_device_reset(hdev, true, false);
241+ hl_device_reset(hdev, HL_RESET_HARD);
230242
231243 out:
232244 return count;
@@ -245,6 +257,9 @@ static ssize_t device_type_show(struct device *dev, · 12 unmodified lines
245257 case ASIC_GAUDI:
246258 str = "GAUDI";
247259 break;
260+ case ASIC_GAUDI_SEC:
261+ str = "GAUDI SEC";
262+ break;
248263 default:
249264 dev_err(hdev->dev, "Unrecognized ASIC type %d\n",
250265 hdev->asic_type);
@@ -344,7 +359,7 @@ static ssize_t eeprom_read_handler(struct file *filp, struct kobject *kobj, · 93 unmodified lines
344359 struct bin_attribute *attr, char *buf, loff_t offset,
345360 size_t max_size)
346361 {
347− struct device *dev = container_of(kobj, struct device, kobj);
362+ struct device *dev = kobj_to_dev(kobj);
348363 struct hl_device *hdev = dev_get_drvdata(dev);
349364 char *data;
350365 int rc;
drivers/misc/habanalabs/gaudi/gaudi.c+314−43
@@ -105,6 +105,36 @@ · 104 unmodified lines
105105
106106 #define GAUDI_PLL_MAX 10
107107
108+/*
109+ * this enum kept here for compatibility with old FW (in which each asic has
110+ * unique PLL numbering
111+ */
112+enum gaudi_pll_index {
113+ GAUDI_CPU_PLL = 0,
114+ GAUDI_PCI_PLL,
115+ GAUDI_SRAM_PLL,
116+ GAUDI_HBM_PLL,
117+ GAUDI_NIC_PLL,
118+ GAUDI_DMA_PLL,
119+ GAUDI_MESH_PLL,
120+ GAUDI_MME_PLL,
121+ GAUDI_TPC_PLL,
122+ GAUDI_IF_PLL,
123+};
124+
125+static enum pll_index gaudi_pll_map[PLL_MAX] = {
126+ [CPU_PLL] = GAUDI_CPU_PLL,
127+ [PCI_PLL] = GAUDI_PCI_PLL,
128+ [SRAM_PLL] = GAUDI_SRAM_PLL,
129+ [HBM_PLL] = GAUDI_HBM_PLL,
130+ [NIC_PLL] = GAUDI_NIC_PLL,
131+ [DMA_PLL] = GAUDI_DMA_PLL,
132+ [MESH_PLL] = GAUDI_MESH_PLL,
133+ [MME_PLL] = GAUDI_MME_PLL,
134+ [TPC_PLL] = GAUDI_TPC_PLL,
135+ [IF_PLL] = GAUDI_IF_PLL,
136+};
137+
108138 static const char gaudi_irq_name[GAUDI_MSI_ENTRIES][GAUDI_MAX_STRING_LEN] = {
109139 "gaudi cq 0_0", "gaudi cq 0_1", "gaudi cq 0_2", "gaudi cq 0_3",
110140 "gaudi cq 1_0", "gaudi cq 1_1", "gaudi cq 1_2", "gaudi cq 1_3",
@@ -396,6 +426,19 @@ get_collective_mode(struct hl_device *hdev, u32 queue_id) · 285 unmodified lines
396426 return HL_COLLECTIVE_NOT_SUPPORTED;
397427 }
398428
429+static inline void set_default_power_values(struct hl_device *hdev)
430+{
431+ struct asic_fixed_properties *prop = &hdev->asic_prop;
432+
433+ if (hdev->card_type == cpucp_card_type_pmc) {
434+ prop->max_power_default = MAX_POWER_DEFAULT_PMC;
435+ prop->dc_power_default = DC_POWER_DEFAULT_PMC;
436+ } else {
437+ prop->max_power_default = MAX_POWER_DEFAULT_PCI;
438+ prop->dc_power_default = DC_POWER_DEFAULT_PCI;
439+ }
440+}
441+
399442 static int gaudi_get_fixed_properties(struct hl_device *hdev)
400443 {
401444 struct asic_fixed_properties *prop = &hdev->asic_prop;
@@ -507,7 +550,7 @@ static int gaudi_get_fixed_properties(struct hl_device *hdev) · 105 unmodified lines
507550 prop->num_of_events = GAUDI_EVENT_SIZE;
508551 prop->tpc_enabled_mask = TPC_ENABLED_MASK;
509552
510− prop->max_power_default = MAX_POWER_DEFAULT_PCI;
553+ set_default_power_values(hdev);
511554
512555 prop->cb_pool_cb_cnt = GAUDI_CB_POOL_CB_CNT;
513556 prop->cb_pool_cb_size = GAUDI_CB_POOL_CB_SIZE;
@@ -532,8 +575,6 @@ static int gaudi_get_fixed_properties(struct hl_device *hdev) · 18 unmodified lines
532575 for (i = 0 ; i < HL_MAX_DCORES ; i++)
533576 prop->first_available_cq[i] = USHRT_MAX;
534577
535− /* disable fw security for now, set it in a later stage */
536− prop->fw_security_disabled = true;
537578 prop->fw_security_status_valid = false;
538579 prop->hard_reset_done_by_fw = false;
539580
@@ -588,6 +629,11 @@ static int gaudi_init_iatu(struct hl_device *hdev) · 48 unmodified lines
588629 struct hl_outbound_pci_region outbound_region;
589630 int rc;
590631
632+ if (hdev->asic_prop.iatu_done_by_fw) {
633+ hdev->asic_funcs->set_dma_mask_from_fw(hdev);
634+ return 0;
635+ }
636+
591637 /* Inbound Region 0 - Bar 0 - Point to SRAM + CFG */
592638 inbound_region.mode = PCI_BAR_MATCH_MODE;
593639 inbound_region.bar = SRAM_BAR_ID;
@@ -632,6 +678,7 @@ static int gaudi_early_init(struct hl_device *hdev) · 38 unmodified lines
632678 {
633679 struct asic_fixed_properties *prop = &hdev->asic_prop;
634680 struct pci_dev *pdev = hdev->pdev;
681+ u32 fw_boot_status;
635682 int rc;
636683
637684 rc = gaudi_get_fixed_properties(hdev);
@@ -665,6 +712,23 @@ static int gaudi_early_init(struct hl_device *hdev) · 27 unmodified lines
665712
666713 prop->dram_pci_bar_size = pci_resource_len(pdev, HBM_BAR_ID);
667714
715+ /* If FW security is enabled at this point it means no access to ELBI */
716+ if (!hdev->asic_prop.fw_security_disabled) {
717+ hdev->asic_prop.iatu_done_by_fw = true;
718+ goto pci_init;
719+ }
720+
721+ rc = hl_pci_elbi_read(hdev, CFG_BASE + mmCPU_BOOT_DEV_STS0,
722+ &fw_boot_status);
723+ if (rc)
724+ goto free_queue_props;
725+
726+ /* Check whether FW is configuring iATU */
727+ if ((fw_boot_status & CPU_BOOT_DEV_STS0_ENABLED) &&
728+ (fw_boot_status & CPU_BOOT_DEV_STS0_FW_IATU_CONF_EN))
729+ hdev->asic_prop.iatu_done_by_fw = true;
730+
731+pci_init:
668732 rc = hl_pci_init(hdev);
669733 if (rc)
670734 goto free_queue_props;
@@ -1588,6 +1652,9 @@ static int gaudi_sw_init(struct hl_device *hdev) · 917 unmodified lines
15881652
15891653 hdev->asic_specific = gaudi;
15901654
1655+ /* store legacy PLL map */
1656+ hdev->legacy_pll_map = gaudi_pll_map;
1657+
15911658 /* Create DMA pool for small allocations */
15921659 hdev->dma_pool = dma_pool_create(dev_name(hdev->dev),
15931660 &hdev->pdev->dev, GAUDI_DMA_POOL_BLK_SIZE, 8, 0);
@@ -1766,8 +1833,7 @@ static int gaudi_enable_msi(struct hl_device *hdev) · 172 unmodified lines
17661833 if (gaudi->hw_cap_initialized & HW_CAP_MSI)
17671834 return 0;
17681835
1769− rc = pci_alloc_irq_vectors(hdev->pdev, 1, GAUDI_MSI_ENTRIES,
1770− PCI_IRQ_MSI);
1836+ rc = pci_alloc_irq_vectors(hdev->pdev, 1, 1, PCI_IRQ_MSI);
17711837 if (rc < 0) {
17721838 dev_err(hdev->dev, "MSI: Failed to enable support %d\n", rc);
17731839 return rc;
@@ -3701,7 +3767,7 @@ static int gaudi_init_cpu(struct hl_device *hdev) · 1927 unmodified lines
37013767 struct gaudi_device *gaudi = hdev->asic_specific;
37023768 int rc;
37033769
3704− if (!hdev->cpu_enable)
3770+ if (!(hdev->fw_components & FW_TYPE_PREBOOT_CPU))
37053771 return 0;
37063772
37073773 if (gaudi->hw_cap_initialized & HW_CAP_CPU)
@@ -4873,7 +4939,7 @@ static int gaudi_pin_memory_before_cs(struct hl_device *hdev, · 1165 unmodified lines
48734939 parser->job_userptr_list, &userptr))
48744940 goto already_pinned;
48754941
4876− userptr = kzalloc(sizeof(*userptr), GFP_ATOMIC);
4942+ userptr = kzalloc(sizeof(*userptr), GFP_KERNEL);
48774943 if (!userptr)
48784944 return -ENOMEM;
48794945
@@ -5684,18 +5750,26 @@ release_cb: · 804 unmodified lines
56845750 static int gaudi_schedule_register_memset(struct hl_device *hdev,
56855751 u32 hw_queue_id, u64 reg_base, u32 num_regs, u32 val)
56865752 {
5687− struct hl_ctx *ctx = hdev->compute_ctx;
5753+ struct hl_ctx *ctx;
56885754 struct hl_pending_cb *pending_cb;
56895755 struct packet_msg_long *pkt;
56905756 u32 cb_size, ctl;
56915757 struct hl_cb *cb;
5692− int i;
5758+ int i, rc;
5759+
5760+ mutex_lock(&hdev->fpriv_list_lock);
5761+ ctx = hdev->compute_ctx;
56935762
56945763 /* If no compute context available or context is going down
56955764 * memset registers directly
56965765 */
5697− if (!ctx || kref_read(&ctx->refcount) == 0)
5698− return gaudi_memset_registers(hdev, reg_base, num_regs, val);
5766+ if (!ctx || kref_read(&ctx->refcount) == 0) {
5767+ rc = gaudi_memset_registers(hdev, reg_base, num_regs, val);
5768+ mutex_unlock(&hdev->fpriv_list_lock);
5769+ return rc;
5770+ }
5771+
5772+ mutex_unlock(&hdev->fpriv_list_lock);
56995773
57005774 cb_size = (sizeof(*pkt) * num_regs) +
57015775 sizeof(struct packet_msg_prot) * 2;
@@ -5911,13 +5985,16 @@ static void gaudi_restore_phase_topology(struct hl_device *hdev) · 209 unmodified lines
59115985
59125986 }
59135987
5914−static int gaudi_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val)
5988+static int gaudi_debugfs_read32(struct hl_device *hdev, u64 addr,
5989+ bool user_address, u32 *val)
59155990 {
59165991 struct asic_fixed_properties *prop = &hdev->asic_prop;
59175992 struct gaudi_device *gaudi = hdev->asic_specific;
5918− u64 hbm_bar_addr;
5993+ u64 hbm_bar_addr, host_phys_end;
59195994 int rc = 0;
59205995
5996+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
5997+
59215998 if ((addr >= CFG_BASE) && (addr < CFG_BASE + CFG_SIZE)) {
59225999
59236000 if ((gaudi->hw_cap_initialized & HW_CAP_CLK_GATE) &&
@@ -5949,6 +6026,9 @@ static int gaudi_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val) · 25 unmodified lines
59496026 }
59506027 if (hbm_bar_addr == U64_MAX)
59516028 rc = -EIO;
6029+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
6030+ user_address && !iommu_present(&pci_bus_type)) {
6031+ *val = *(u32 *) phys_to_virt(addr - HOST_PHYS_BASE);
59526032 } else {
59536033 rc = -EFAULT;
59546034 }
@@ -5956,13 +6036,16 @@ static int gaudi_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val) · 1 unmodified line
59566036 return rc;
59576037 }
59586038
5959−static int gaudi_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val)
6039+static int gaudi_debugfs_write32(struct hl_device *hdev, u64 addr,
6040+ bool user_address, u32 val)
59606041 {
59616042 struct asic_fixed_properties *prop = &hdev->asic_prop;
59626043 struct gaudi_device *gaudi = hdev->asic_specific;
5963− u64 hbm_bar_addr;
6044+ u64 hbm_bar_addr, host_phys_end;
59646045 int rc = 0;
59656046
6047+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
6048+
59666049 if ((addr >= CFG_BASE) && (addr < CFG_BASE + CFG_SIZE)) {
59676050
59686051 if ((gaudi->hw_cap_initialized & HW_CAP_CLK_GATE) &&
@@ -5994,6 +6077,9 @@ static int gaudi_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val) · 25 unmodified lines
59946077 }
59956078 if (hbm_bar_addr == U64_MAX)
59966079 rc = -EIO;
6080+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
6081+ user_address && !iommu_present(&pci_bus_type)) {
6082+ *(u32 *) phys_to_virt(addr - HOST_PHYS_BASE) = val;
59976083 } else {
59986084 rc = -EFAULT;
59996085 }
@@ -6001,13 +6087,16 @@ static int gaudi_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val) · 1 unmodified line
60016087 return rc;
60026088 }
60036089
6004−static int gaudi_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val)
6090+static int gaudi_debugfs_read64(struct hl_device *hdev, u64 addr,
6091+ bool user_address, u64 *val)
60056092 {
60066093 struct asic_fixed_properties *prop = &hdev->asic_prop;
60076094 struct gaudi_device *gaudi = hdev->asic_specific;
6008− u64 hbm_bar_addr;
6095+ u64 hbm_bar_addr, host_phys_end;
60096096 int rc = 0;
60106097
6098+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
6099+
60116100 if ((addr >= CFG_BASE) && (addr <= CFG_BASE + CFG_SIZE - sizeof(u64))) {
60126101
60136102 if ((gaudi->hw_cap_initialized & HW_CAP_CLK_GATE) &&
@@ -6043,6 +6132,9 @@ static int gaudi_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val) · 29 unmodified lines
60436132 }
60446133 if (hbm_bar_addr == U64_MAX)
60456134 rc = -EIO;
6135+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
6136+ user_address && !iommu_present(&pci_bus_type)) {
6137+ *val = *(u64 *) phys_to_virt(addr - HOST_PHYS_BASE);
60466138 } else {
60476139 rc = -EFAULT;
60486140 }
@@ -6050,13 +6142,16 @@ static int gaudi_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val) · 1 unmodified line
60506142 return rc;
60516143 }
60526144
6053−static int gaudi_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val)
6145+static int gaudi_debugfs_write64(struct hl_device *hdev, u64 addr,
6146+ bool user_address, u64 val)
60546147 {
60556148 struct asic_fixed_properties *prop = &hdev->asic_prop;
60566149 struct gaudi_device *gaudi = hdev->asic_specific;
6057− u64 hbm_bar_addr;
6150+ u64 hbm_bar_addr, host_phys_end;
60586151 int rc = 0;
60596152
6153+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
6154+
60606155 if ((addr >= CFG_BASE) && (addr <= CFG_BASE + CFG_SIZE - sizeof(u64))) {
60616156
60626157 if ((gaudi->hw_cap_initialized & HW_CAP_CLK_GATE) &&
@@ -6091,6 +6186,9 @@ static int gaudi_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val) · 28 unmodified lines
60916186 }
60926187 if (hbm_bar_addr == U64_MAX)
60936188 rc = -EIO;
6189+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
6190+ user_address && !iommu_present(&pci_bus_type)) {
6191+ *(u64 *) phys_to_virt(addr - HOST_PHYS_BASE) = val;
60946192 } else {
60956193 rc = -EFAULT;
60966194 }
@@ -6098,6 +6196,164 @@ static int gaudi_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val) · 1 unmodified line
60986196 return rc;
60996197 }
61006198
6199+static int gaudi_dma_core_transfer(struct hl_device *hdev, int dma_id, u64 addr,
6200+ u32 size_to_dma, dma_addr_t dma_addr)
6201+{
6202+ u32 err_cause, val;
6203+ u64 dma_offset;
6204+ int rc;
6205+
6206+ dma_offset = dma_id * DMA_CORE_OFFSET;
6207+
6208+ WREG32(mmDMA0_CORE_SRC_BASE_LO + dma_offset, lower_32_bits(addr));
6209+ WREG32(mmDMA0_CORE_SRC_BASE_HI + dma_offset, upper_32_bits(addr));
6210+ WREG32(mmDMA0_CORE_DST_BASE_LO + dma_offset, lower_32_bits(dma_addr));
6211+ WREG32(mmDMA0_CORE_DST_BASE_HI + dma_offset, upper_32_bits(dma_addr));
6212+ WREG32(mmDMA0_CORE_DST_TSIZE_0 + dma_offset, size_to_dma);
6213+ WREG32(mmDMA0_CORE_COMMIT + dma_offset,
6214+ (1 << DMA0_CORE_COMMIT_LIN_SHIFT));
6215+
6216+ rc = hl_poll_timeout(
6217+ hdev,
6218+ mmDMA0_CORE_STS0 + dma_offset,
6219+ val,
6220+ ((val & DMA0_CORE_STS0_BUSY_MASK) == 0),
6221+ 0,
6222+ 1000000);
6223+
6224+ if (rc) {
6225+ dev_err(hdev->dev,
6226+ "DMA %d timed-out during reading of 0x%llx\n",
6227+ dma_id, addr);
6228+ return -EIO;
6229+ }
6230+
6231+ /* Verify DMA is OK */
6232+ err_cause = RREG32(mmDMA0_CORE_ERR_CAUSE + dma_offset);
6233+ if (err_cause) {
6234+ dev_err(hdev->dev, "DMA Failed, cause 0x%x\n", err_cause);
6235+ dev_dbg(hdev->dev,
6236+ "Clearing DMA0 engine from errors (cause 0x%x)\n",
6237+ err_cause);
6238+ WREG32(mmDMA0_CORE_ERR_CAUSE + dma_offset, err_cause);
6239+
6240+ return -EIO;
6241+ }
6242+
6243+ return 0;
6244+}
6245+
6246+static int gaudi_debugfs_read_dma(struct hl_device *hdev, u64 addr, u32 size,
6247+ void *blob_addr)
6248+{
6249+ u32 dma_core_sts0, err_cause, cfg1, size_left, pos, size_to_dma;
6250+ struct gaudi_device *gaudi = hdev->asic_specific;
6251+ u64 dma_offset, qm_offset;
6252+ dma_addr_t dma_addr;
6253+ void *kernel_addr;
6254+ bool is_eng_idle;
6255+ int rc, dma_id;
6256+
6257+ kernel_addr = hdev->asic_funcs->asic_dma_alloc_coherent(
6258+ hdev, SZ_2M,
6259+ &dma_addr,
6260+ GFP_KERNEL | __GFP_ZERO);
6261+
6262+ if (!kernel_addr)
6263+ return -ENOMEM;
6264+
6265+ mutex_lock(&gaudi->clk_gate_mutex);
6266+
6267+ hdev->asic_funcs->disable_clock_gating(hdev);
6268+
6269+ hdev->asic_funcs->hw_queues_lock(hdev);
6270+
6271+ dma_id = gaudi_dma_assignment[GAUDI_PCI_DMA_1];
6272+ dma_offset = dma_id * DMA_CORE_OFFSET;
6273+ qm_offset = dma_id * DMA_QMAN_OFFSET;
6274+ dma_core_sts0 = RREG32(mmDMA0_CORE_STS0 + dma_offset);
6275+ is_eng_idle = IS_DMA_IDLE(dma_core_sts0);
6276+
6277+ if (!is_eng_idle) {
6278+ dma_id = gaudi_dma_assignment[GAUDI_PCI_DMA_2];
6279+ dma_offset = dma_id * DMA_CORE_OFFSET;
6280+ qm_offset = dma_id * DMA_QMAN_OFFSET;
6281+ dma_core_sts0 = RREG32(mmDMA0_CORE_STS0 + dma_offset);
6282+ is_eng_idle = IS_DMA_IDLE(dma_core_sts0);
6283+
6284+ if (!is_eng_idle) {
6285+ dev_err_ratelimited(hdev->dev,
6286+ "Can't read via DMA because it is BUSY\n");
6287+ rc = -EAGAIN;
6288+ goto out;
6289+ }
6290+ }
6291+
6292+ cfg1 = RREG32(mmDMA0_QM_GLBL_CFG1 + qm_offset);
6293+ WREG32(mmDMA0_QM_GLBL_CFG1 + qm_offset,
6294+ 0xF << DMA0_QM_GLBL_CFG1_CP_STOP_SHIFT);
6295+
6296+ /* TODO: remove this by mapping the DMA temporary buffer to the MMU
6297+ * using the compute ctx ASID, if exists. If not, use the kernel ctx
6298+ * ASID
6299+ */
6300+ WREG32_OR(mmDMA0_CORE_PROT + dma_offset, BIT(DMA0_CORE_PROT_VAL_SHIFT));
6301+
6302+ /* Verify DMA is OK */
6303+ err_cause = RREG32(mmDMA0_CORE_ERR_CAUSE + dma_offset);
6304+ if (err_cause) {
6305+ dev_dbg(hdev->dev,
6306+ "Clearing DMA0 engine from errors (cause 0x%x)\n",
6307+ err_cause);
6308+ WREG32(mmDMA0_CORE_ERR_CAUSE + dma_offset, err_cause);
6309+ }
6310+
6311+ pos = 0;
6312+ size_left = size;
6313+ size_to_dma = SZ_2M;
6314+
6315+ while (size_left > 0) {
6316+
6317+ if (size_left < SZ_2M)
6318+ size_to_dma = size_left;
6319+
6320+ rc = gaudi_dma_core_transfer(hdev, dma_id, addr, size_to_dma,
6321+ dma_addr);
6322+ if (rc)
6323+ break;
6324+
6325+ memcpy(blob_addr + pos, kernel_addr, size_to_dma);
6326+
6327+ if (size_left <= SZ_2M)
6328+ break;
6329+
6330+ pos += SZ_2M;
6331+ addr += SZ_2M;
6332+ size_left -= SZ_2M;
6333+ }
6334+
6335+ /* TODO: remove this by mapping the DMA temporary buffer to the MMU
6336+ * using the compute ctx ASID, if exists. If not, use the kernel ctx
6337+ * ASID
6338+ */
6339+ WREG32_AND(mmDMA0_CORE_PROT + dma_offset,
6340+ ~BIT(DMA0_CORE_PROT_VAL_SHIFT));
6341+
6342+ WREG32(mmDMA0_QM_GLBL_CFG1 + qm_offset, cfg1);
6343+
6344+out:
6345+ hdev->asic_funcs->hw_queues_unlock(hdev);
6346+
6347+ hdev->asic_funcs->set_clock_gating(hdev);
6348+
6349+ mutex_unlock(&gaudi->clk_gate_mutex);
6350+
6351+ hdev->asic_funcs->asic_dma_free_coherent(hdev, SZ_2M, kernel_addr,
6352+ dma_addr);
6353+
6354+ return rc;
6355+}
6356+
61016357 static u64 gaudi_read_pte(struct hl_device *hdev, u64 addr)
61026358 {
61036359 struct gaudi_device *gaudi = hdev->asic_specific;
@@ -6851,7 +7107,8 @@ static void gaudi_handle_qman_err_generic(struct hl_device *hdev, · 747 unmodified lines
68517107 }
68527108
68537109 /* Write 1 clear errors */
6854− WREG32(glbl_sts_addr + 4 * i, glbl_sts_clr_val);
7110+ if (!hdev->stop_on_err)
7111+ WREG32(glbl_sts_addr + 4 * i, glbl_sts_clr_val);
68557112 }
68567113
68577114 arb_err_val = RREG32(arb_err_addr);
@@ -7097,6 +7354,15 @@ static void gaudi_print_irq_info(struct hl_device *hdev, u16 event_type, · 239 unmodified lines
70977354 }
70987355 }
70997356
7357+static void gaudi_print_out_of_sync_info(struct hl_device *hdev,
7358+ struct cpucp_pkt_sync_err *sync_err)
7359+{
7360+ struct hl_hw_queue *q = &hdev->kernel_queues[GAUDI_QUEUE_ID_CPU_PQ];
7361+
7362+ dev_err(hdev->dev, "Out of sync with FW, FW: pi=%u, ci=%u, LKD: pi=%u, ci=%u\n",
7363+ sync_err->pi, sync_err->ci, q->pi, atomic_read(&q->ci));
7364+}
7365+
71007366 static int gaudi_soft_reset_late_init(struct hl_device *hdev)
71017367 {
71027368 struct gaudi_device *gaudi = hdev->asic_specific;
@@ -7371,18 +7637,14 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 268 unmodified lines
73717637 case GAUDI_EVENT_MMU_DERR:
73727638 gaudi_print_irq_info(hdev, event_type, true);
73737639 gaudi_handle_ecc_event(hdev, event_type, &eq_entry->ecc_data);
7374− if (hdev->hard_reset_on_fw_events)
7375− hl_device_reset(hdev, true, false);
7376− break;
7640+ goto reset_device;
73777641
73787642 case GAUDI_EVENT_GIC500:
73797643 case GAUDI_EVENT_AXI_ECC:
73807644 case GAUDI_EVENT_L2_RAM_ECC:
73817645 case GAUDI_EVENT_PLL0 ... GAUDI_EVENT_PLL17:
73827646 gaudi_print_irq_info(hdev, event_type, false);
7383− if (hdev->hard_reset_on_fw_events)
7384− hl_device_reset(hdev, true, false);
7385− break;
7647+ goto reset_device;
73867648
73877649 case GAUDI_EVENT_HBM0_SPI_0:
73887650 case GAUDI_EVENT_HBM1_SPI_0:
@@ -7392,9 +7654,7 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 3 unmodified lines
73927654 gaudi_hbm_read_interrupts(hdev,
73937655 gaudi_hbm_event_to_dev(event_type),
73947656 &eq_entry->hbm_ecc_data);
7395− if (hdev->hard_reset_on_fw_events)
7396− hl_device_reset(hdev, true, false);
7397− break;
7657+ goto reset_device;
73987658
73997659 case GAUDI_EVENT_HBM0_SPI_1:
74007660 case GAUDI_EVENT_HBM1_SPI_1:
@@ -7423,8 +7683,7 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 22 unmodified lines
74237683 dev_err(hdev->dev, "hard reset required due to %s\n",
74247684 gaudi_irq_map_table[event_type].name);
74257685
7426− if (hdev->hard_reset_on_fw_events)
7427− hl_device_reset(hdev, true, false);
7686+ goto reset_device;
74287687 } else {
74297688 hl_fw_unmask_irq(hdev, event_type);
74307689 }
@@ -7446,8 +7705,7 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 15 unmodified lines
74467705 dev_err(hdev->dev, "hard reset required due to %s\n",
74477706 gaudi_irq_map_table[event_type].name);
74487707
7449− if (hdev->hard_reset_on_fw_events)
7450− hl_device_reset(hdev, true, false);
7708+ goto reset_device;
74517709 } else {
74527710 hl_fw_unmask_irq(hdev, event_type);
74537711 }
@@ -7516,9 +7774,7 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 62 unmodified lines
75167774
75177775 case GAUDI_EVENT_RAZWI_OR_ADC_SW:
75187776 gaudi_print_irq_info(hdev, event_type, true);
7519− if (hdev->hard_reset_on_fw_events)
7520− hl_device_reset(hdev, true, false);
7521− break;
7777+ goto reset_device;
75227778
75237779 case GAUDI_EVENT_TPC0_BMON_SPMU:
75247780 case GAUDI_EVENT_TPC1_BMON_SPMU:
@@ -7552,11 +7808,28 @@ static void gaudi_handle_eqe(struct hl_device *hdev, · 27 unmodified lines
75527808 event_type, cause);
75537809 break;
75547810
7811+ case GAUDI_EVENT_DEV_RESET_REQ:
7812+ gaudi_print_irq_info(hdev, event_type, false);
7813+ goto reset_device;
7814+
7815+ case GAUDI_EVENT_PKT_QUEUE_OUT_SYNC:
7816+ gaudi_print_irq_info(hdev, event_type, false);
7817+ gaudi_print_out_of_sync_info(hdev, &eq_entry->pkt_sync_err);
7818+ goto reset_device;
7819+
75557820 default:
75567821 dev_err(hdev->dev, "Received invalid H/W interrupt %d\n",
75577822 event_type);
75587823 break;
75597824 }
7825+
7826+ return;
7827+
7828+reset_device:
7829+ if (hdev->hard_reset_on_fw_events)
7830+ hl_device_reset(hdev, HL_RESET_HARD);
7831+ else
7832+ hl_fw_unmask_irq(hdev, event_type);
75607833 }
75617834
75627835 static void *gaudi_get_events_stat(struct hl_device *hdev, bool aggregate,
@@ -7607,7 +7880,7 @@ static int gaudi_mmu_invalidate_cache(struct hl_device *hdev, bool is_hard, · 44 unmodified lines
76077880 if (rc) {
76087881 dev_err_ratelimited(hdev->dev,
76097882 "MMU cache invalidation timeout\n");
7610− hl_device_reset(hdev, true, false);
7883+ hl_device_reset(hdev, HL_RESET_HARD);
76117884 }
76127885
76137886 return rc;
@@ -7656,7 +7929,7 @@ static int gaudi_mmu_invalidate_cache_range(struct hl_device *hdev, · 42 unmodified lines
76567929 if (rc) {
76577930 dev_err_ratelimited(hdev->dev,
76587931 "MMU cache invalidation timeout\n");
7659− hl_device_reset(hdev, true, false);
7932+ hl_device_reset(hdev, HL_RESET_HARD);
76607933 }
76617934
76627935 return rc;
@@ -7714,7 +7987,7 @@ static int gaudi_cpucp_info_get(struct hl_device *hdev) · 51 unmodified lines
77147987 if (!(gaudi->hw_cap_initialized & HW_CAP_CPU_Q))
77157988 return 0;
77167989
7717− rc = hl_fw_cpucp_info_get(hdev, mmCPU_BOOT_DEV_STS0, mmCPU_BOOT_ERR0);
7990+ rc = hl_fw_cpucp_handshake(hdev, mmCPU_BOOT_DEV_STS0, mmCPU_BOOT_ERR0);
77187991 if (rc)
77197992 return rc;
77207993
@@ -7724,10 +7997,7 @@ static int gaudi_cpucp_info_get(struct hl_device *hdev) · 3 unmodified lines
77247997
77257998 hdev->card_type = le32_to_cpu(hdev->asic_prop.cpucp_info.card_type);
77267999
7727− if (hdev->card_type == cpucp_card_type_pci)
7728− prop->max_power_default = MAX_POWER_DEFAULT_PCI;
7729− else if (hdev->card_type == cpucp_card_type_pmc)
7730− prop->max_power_default = MAX_POWER_DEFAULT_PMC;
8000+ set_default_power_values(hdev);
77318001
77328002 hdev->max_power = prop->max_power_default;
77338003
@@ -8549,6 +8819,7 @@ static const struct hl_asic_funcs gaudi_funcs = { · 815 unmodified lines
85498819 .debugfs_write32 = gaudi_debugfs_write32,
85508820 .debugfs_read64 = gaudi_debugfs_read64,
85518821 .debugfs_write64 = gaudi_debugfs_write64,
8822+ .debugfs_read_dma = gaudi_debugfs_read_dma,
85528823 .add_device_attr = gaudi_add_device_attr,
85538824 .handle_eqe = gaudi_handle_eqe,
85548825 .set_pll_profile = gaudi_set_pll_profile,
drivers/misc/habanalabs/gaudi/gaudiP.h+3−0
@@ -47,6 +47,9 @@ · 46 unmodified lines
4747 #define MAX_POWER_DEFAULT_PCI 200000 /* 200W */
4848 #define MAX_POWER_DEFAULT_PMC 350000 /* 350W */
4949
50+#define DC_POWER_DEFAULT_PCI 60000 /* 60W */
51+#define DC_POWER_DEFAULT_PMC 60000 /* 60W */
52+
5053 #define GAUDI_CPU_TIMEOUT_USEC 30000000 /* 30s */
5154
5255 #define TPC_ENABLED_MASK 0xFF
drivers/misc/habanalabs/gaudi/gaudi_security.c+0−8
@@ -9556,7 +9556,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 9555 unmodified lines
95569556 mask = 1U << ((mmTPC0_CFG_PROT & 0x7F) >> 2);
95579557 mask |= 1U << ((mmTPC0_CFG_VFLAGS & 0x7F) >> 2);
95589558 mask |= 1U << ((mmTPC0_CFG_SFLAGS & 0x7F) >> 2);
9559− mask |= 1U << ((mmTPC0_CFG_STATUS & 0x7F) >> 2);
95609559 mask |= 1U << ((mmTPC0_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
95619560 mask |= 1U << ((mmTPC0_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
95629561 mask |= 1U << ((mmTPC0_CFG_TPC_STALL & 0x7F) >> 2);
@@ -10011,7 +10010,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 448 unmodified lines
1001110010 mask = 1U << ((mmTPC1_CFG_PROT & 0x7F) >> 2);
1001210011 mask |= 1U << ((mmTPC1_CFG_VFLAGS & 0x7F) >> 2);
1001310012 mask |= 1U << ((mmTPC1_CFG_SFLAGS & 0x7F) >> 2);
10014− mask |= 1U << ((mmTPC1_CFG_STATUS & 0x7F) >> 2);
1001510013 mask |= 1U << ((mmTPC1_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1001610014 mask |= 1U << ((mmTPC1_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1001710015 mask |= 1U << ((mmTPC1_CFG_TPC_STALL & 0x7F) >> 2);
@@ -10465,7 +10463,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 447 unmodified lines
1046510463 mask = 1U << ((mmTPC2_CFG_PROT & 0x7F) >> 2);
1046610464 mask |= 1U << ((mmTPC2_CFG_VFLAGS & 0x7F) >> 2);
1046710465 mask |= 1U << ((mmTPC2_CFG_SFLAGS & 0x7F) >> 2);
10468− mask |= 1U << ((mmTPC2_CFG_STATUS & 0x7F) >> 2);
1046910466 mask |= 1U << ((mmTPC2_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1047010467 mask |= 1U << ((mmTPC2_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1047110468 mask |= 1U << ((mmTPC2_CFG_TPC_STALL & 0x7F) >> 2);
@@ -10919,7 +10916,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 447 unmodified lines
1091910916 mask = 1U << ((mmTPC3_CFG_PROT & 0x7F) >> 2);
1092010917 mask |= 1U << ((mmTPC3_CFG_VFLAGS & 0x7F) >> 2);
1092110918 mask |= 1U << ((mmTPC3_CFG_SFLAGS & 0x7F) >> 2);
10922− mask |= 1U << ((mmTPC3_CFG_STATUS & 0x7F) >> 2);
1092310919 mask |= 1U << ((mmTPC3_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1092410920 mask |= 1U << ((mmTPC3_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1092510921 mask |= 1U << ((mmTPC3_CFG_TPC_STALL & 0x7F) >> 2);
@@ -11373,7 +11369,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 447 unmodified lines
1137311369 mask = 1U << ((mmTPC4_CFG_PROT & 0x7F) >> 2);
1137411370 mask |= 1U << ((mmTPC4_CFG_VFLAGS & 0x7F) >> 2);
1137511371 mask |= 1U << ((mmTPC4_CFG_SFLAGS & 0x7F) >> 2);
11376− mask |= 1U << ((mmTPC4_CFG_STATUS & 0x7F) >> 2);
1137711372 mask |= 1U << ((mmTPC4_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1137811373 mask |= 1U << ((mmTPC4_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1137911374 mask |= 1U << ((mmTPC4_CFG_TPC_STALL & 0x7F) >> 2);
@@ -11827,7 +11822,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 447 unmodified lines
1182711822 mask = 1U << ((mmTPC5_CFG_PROT & 0x7F) >> 2);
1182811823 mask |= 1U << ((mmTPC5_CFG_VFLAGS & 0x7F) >> 2);
1182911824 mask |= 1U << ((mmTPC5_CFG_SFLAGS & 0x7F) >> 2);
11830− mask |= 1U << ((mmTPC5_CFG_STATUS & 0x7F) >> 2);
1183111825 mask |= 1U << ((mmTPC5_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1183211826 mask |= 1U << ((mmTPC5_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1183311827 mask |= 1U << ((mmTPC5_CFG_TPC_STALL & 0x7F) >> 2);
@@ -12283,7 +12277,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 449 unmodified lines
1228312277 mask = 1U << ((mmTPC6_CFG_PROT & 0x7F) >> 2);
1228412278 mask |= 1U << ((mmTPC6_CFG_VFLAGS & 0x7F) >> 2);
1228512279 mask |= 1U << ((mmTPC6_CFG_SFLAGS & 0x7F) >> 2);
12286− mask |= 1U << ((mmTPC6_CFG_STATUS & 0x7F) >> 2);
1228712280 mask |= 1U << ((mmTPC6_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1228812281 mask |= 1U << ((mmTPC6_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1228912282 mask |= 1U << ((mmTPC6_CFG_TPC_STALL & 0x7F) >> 2);
@@ -12739,7 +12732,6 @@ static void gaudi_init_tpc_protection_bits(struct hl_device *hdev) · 449 unmodified lines
1273912732 mask = 1U << ((mmTPC7_CFG_PROT & 0x7F) >> 2);
1274012733 mask |= 1U << ((mmTPC7_CFG_VFLAGS & 0x7F) >> 2);
1274112734 mask |= 1U << ((mmTPC7_CFG_SFLAGS & 0x7F) >> 2);
12742− mask |= 1U << ((mmTPC7_CFG_STATUS & 0x7F) >> 2);
1274312735 mask |= 1U << ((mmTPC7_CFG_CFG_BASE_ADDRESS_HIGH & 0x7F) >> 2);
1274412736 mask |= 1U << ((mmTPC7_CFG_CFG_SUBTRACT_VALUE & 0x7F) >> 2);
1274512737 mask |= 1U << ((mmTPC7_CFG_TPC_STALL & 0x7F) >> 2);
drivers/misc/habanalabs/goya/goya.c+123−17
@@ -118,6 +118,29 @@ · 117 unmodified lines
118118 #define IS_MME_IDLE(mme_arch_sts) \
119119 (((mme_arch_sts) & MME_ARCH_IDLE_MASK) == MME_ARCH_IDLE_MASK)
120120
121+/*
122+ * this enum kept here for compatibility with old FW (in which each asic has
123+ * unique PLL numbering
124+ */
125+enum goya_pll_index {
126+ GOYA_CPU_PLL = 0,
127+ GOYA_IC_PLL,
128+ GOYA_MC_PLL,
129+ GOYA_MME_PLL,
130+ GOYA_PCI_PLL,
131+ GOYA_EMMC_PLL,
132+ GOYA_TPC_PLL,
133+};
134+
135+static enum pll_index goya_pll_map[PLL_MAX] = {
136+ [CPU_PLL] = GOYA_CPU_PLL,
137+ [IC_PLL] = GOYA_IC_PLL,
138+ [MC_PLL] = GOYA_MC_PLL,
139+ [MME_PLL] = GOYA_MME_PLL,
140+ [PCI_PLL] = GOYA_PCI_PLL,
141+ [EMMC_PLL] = GOYA_EMMC_PLL,
142+ [TPC_PLL] = GOYA_TPC_PLL,
143+};
121144
122145 static const char goya_irq_name[GOYA_MSIX_ENTRIES][GOYA_MAX_STRING_LEN] = {
123146 "goya cq 0", "goya cq 1", "goya cq 2", "goya cq 3",
@@ -446,6 +469,7 @@ int goya_get_fixed_properties(struct hl_device *hdev) · 322 unmodified lines
446469 prop->cb_pool_cb_cnt = GOYA_CB_POOL_CB_CNT;
447470 prop->cb_pool_cb_size = GOYA_CB_POOL_CB_SIZE;
448471 prop->max_power_default = MAX_POWER_DEFAULT;
472+ prop->dc_power_default = DC_POWER_DEFAULT;
449473 prop->tpc_enabled_mask = TPC_ENABLED_MASK;
450474 prop->pcie_dbi_base_address = mmPCIE_DBI_BASE;
451475 prop->pcie_aux_dbi_reg_addr = CFG_BASE + mmPCIE_AUX_DBI;
@@ -460,8 +484,6 @@ int goya_get_fixed_properties(struct hl_device *hdev) · 8 unmodified lines
460484 for (i = 0 ; i < HL_MAX_DCORES ; i++)
461485 prop->first_available_cq[i] = USHRT_MAX;
462486
463− /* disable fw security for now, set it in a later stage */
464− prop->fw_security_disabled = true;
465487 prop->fw_security_status_valid = false;
466488 prop->hard_reset_done_by_fw = false;
467489
@@ -533,6 +555,11 @@ static int goya_init_iatu(struct hl_device *hdev) · 65 unmodified lines
533555 struct hl_outbound_pci_region outbound_region;
534556 int rc;
535557
558+ if (hdev->asic_prop.iatu_done_by_fw) {
559+ hdev->asic_funcs->set_dma_mask_from_fw(hdev);
560+ return 0;
561+ }
562+
536563 /* Inbound Region 0 - Bar 0 - Point to SRAM and CFG */
537564 inbound_region.mode = PCI_BAR_MATCH_MODE;
538565 inbound_region.bar = SRAM_CFG_BAR_ID;
@@ -580,7 +607,7 @@ static int goya_early_init(struct hl_device *hdev) · 41 unmodified lines
580607 {
581608 struct asic_fixed_properties *prop = &hdev->asic_prop;
582609 struct pci_dev *pdev = hdev->pdev;
583− u32 val;
610+ u32 fw_boot_status, val;
584611 int rc;
585612
586613 rc = goya_get_fixed_properties(hdev);
@@ -614,6 +641,23 @@ static int goya_early_init(struct hl_device *hdev) · 27 unmodified lines
614641
615642 prop->dram_pci_bar_size = pci_resource_len(pdev, DDR_BAR_ID);
616643
644+ /* If FW security is enabled at this point it means no access to ELBI */
645+ if (!hdev->asic_prop.fw_security_disabled) {
646+ hdev->asic_prop.iatu_done_by_fw = true;
647+ goto pci_init;
648+ }
649+
650+ rc = hl_pci_elbi_read(hdev, CFG_BASE + mmCPU_BOOT_DEV_STS0,
651+ &fw_boot_status);
652+ if (rc)
653+ goto free_queue_props;
654+
655+ /* Check whether FW is configuring iATU */
656+ if ((fw_boot_status & CPU_BOOT_DEV_STS0_ENABLED) &&
657+ (fw_boot_status & CPU_BOOT_DEV_STS0_FW_IATU_CONF_EN))
658+ hdev->asic_prop.iatu_done_by_fw = true;
659+
660+pci_init:
617661 rc = hl_pci_init(hdev);
618662 if (rc)
619663 goto free_queue_props;
@@ -853,6 +897,9 @@ static int goya_sw_init(struct hl_device *hdev) · 233 unmodified lines
853897
854898 hdev->asic_specific = goya;
855899
900+ /* store legacy PLL map */
901+ hdev->legacy_pll_map = goya_pll_map;
902+
856903 /* Create DMA pool for small allocations */
857904 hdev->dma_pool = dma_pool_create(dev_name(hdev->dev),
858905 &hdev->pdev->dev, GOYA_DMA_POOL_BLK_SIZE, 8, 0);
@@ -2429,7 +2476,7 @@ static int goya_init_cpu(struct hl_device *hdev) · 1570 unmodified lines
24292476 struct goya_device *goya = hdev->asic_specific;
24302477 int rc;
24312478
2432− if (!hdev->cpu_enable)
2479+ if (!(hdev->fw_components & FW_TYPE_PREBOOT_CPU))
24332480 return 0;
24342481
24352482 if (goya->hw_cap_initialized & HW_CAP_CPU)
@@ -3221,7 +3268,7 @@ static int goya_pin_memory_before_cs(struct hl_device *hdev, · 785 unmodified lines
32213268 parser->job_userptr_list, &userptr))
32223269 goto already_pinned;
32233270
3224− userptr = kzalloc(sizeof(*userptr), GFP_ATOMIC);
3271+ userptr = kzalloc(sizeof(*userptr), GFP_KERNEL);
32253272 if (!userptr)
32263273 return -ENOMEM;
32273274
@@ -4101,12 +4148,15 @@ static void goya_clear_sm_regs(struct hl_device *hdev) · 873 unmodified lines
41014148 * lead to undefined behavior and therefore, should be done with extreme care
41024149 *
41034150 */
4104−static int goya_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val)
4151+static int goya_debugfs_read32(struct hl_device *hdev, u64 addr,
4152+ bool user_address, u32 *val)
41054153 {
41064154 struct asic_fixed_properties *prop = &hdev->asic_prop;
4107− u64 ddr_bar_addr;
4155+ u64 ddr_bar_addr, host_phys_end;
41084156 int rc = 0;
41094157
4158+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
4159+
41104160 if ((addr >= CFG_BASE) && (addr < CFG_BASE + CFG_SIZE)) {
41114161 *val = RREG32(addr - CFG_BASE);
41124162
@@ -4132,6 +4182,10 @@ static int goya_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val) · 19 unmodified lines
41324182 if (ddr_bar_addr == U64_MAX)
41334183 rc = -EIO;
41344184
4185+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
4186+ user_address && !iommu_present(&pci_bus_type)) {
4187+ *val = *(u32 *) phys_to_virt(addr - HOST_PHYS_BASE);
4188+
41354189 } else {
41364190 rc = -EFAULT;
41374191 }
@@ -4154,12 +4208,15 @@ static int goya_debugfs_read32(struct hl_device *hdev, u64 addr, u32 *val) · 16 unmodified lines
41544208 * lead to undefined behavior and therefore, should be done with extreme care
41554209 *
41564210 */
4157−static int goya_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val)
4211+static int goya_debugfs_write32(struct hl_device *hdev, u64 addr,
4212+ bool user_address, u32 val)
41584213 {
41594214 struct asic_fixed_properties *prop = &hdev->asic_prop;
4160− u64 ddr_bar_addr;
4215+ u64 ddr_bar_addr, host_phys_end;
41614216 int rc = 0;
41624217
4218+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
4219+
41634220 if ((addr >= CFG_BASE) && (addr < CFG_BASE + CFG_SIZE)) {
41644221 WREG32(addr - CFG_BASE, val);
41654222
@@ -4185,6 +4242,10 @@ static int goya_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val) · 19 unmodified lines
41854242 if (ddr_bar_addr == U64_MAX)
41864243 rc = -EIO;
41874244
4245+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
4246+ user_address && !iommu_present(&pci_bus_type)) {
4247+ *(u32 *) phys_to_virt(addr - HOST_PHYS_BASE) = val;
4248+
41884249 } else {
41894250 rc = -EFAULT;
41904251 }
@@ -4192,12 +4253,15 @@ static int goya_debugfs_write32(struct hl_device *hdev, u64 addr, u32 val) · 1 unmodified line
41924253 return rc;
41934254 }
41944255
4195−static int goya_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val)
4256+static int goya_debugfs_read64(struct hl_device *hdev, u64 addr,
4257+ bool user_address, u64 *val)
41964258 {
41974259 struct asic_fixed_properties *prop = &hdev->asic_prop;
4198− u64 ddr_bar_addr;
4260+ u64 ddr_bar_addr, host_phys_end;
41994261 int rc = 0;
42004262
4263+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
4264+
42014265 if ((addr >= CFG_BASE) && (addr <= CFG_BASE + CFG_SIZE - sizeof(u64))) {
42024266 u32 val_l = RREG32(addr - CFG_BASE);
42034267 u32 val_h = RREG32(addr + sizeof(u32) - CFG_BASE);
@@ -4227,6 +4291,10 @@ static int goya_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val) · 23 unmodified lines
42274291 if (ddr_bar_addr == U64_MAX)
42284292 rc = -EIO;
42294293
4294+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
4295+ user_address && !iommu_present(&pci_bus_type)) {
4296+ *val = *(u64 *) phys_to_virt(addr - HOST_PHYS_BASE);
4297+
42304298 } else {
42314299 rc = -EFAULT;
42324300 }
@@ -4234,12 +4302,15 @@ static int goya_debugfs_read64(struct hl_device *hdev, u64 addr, u64 *val) · 1 unmodified line
42344302 return rc;
42354303 }
42364304
4237−static int goya_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val)
4305+static int goya_debugfs_write64(struct hl_device *hdev, u64 addr,
4306+ bool user_address, u64 val)
42384307 {
42394308 struct asic_fixed_properties *prop = &hdev->asic_prop;
4240− u64 ddr_bar_addr;
4309+ u64 ddr_bar_addr, host_phys_end;
42414310 int rc = 0;
42424311
4312+ host_phys_end = HOST_PHYS_BASE + HOST_PHYS_SIZE;
4313+
42434314 if ((addr >= CFG_BASE) && (addr <= CFG_BASE + CFG_SIZE - sizeof(u64))) {
42444315 WREG32(addr - CFG_BASE, lower_32_bits(val));
42454316 WREG32(addr + sizeof(u32) - CFG_BASE, upper_32_bits(val));
@@ -4267,6 +4338,10 @@ static int goya_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val) · 21 unmodified lines
42674338 if (ddr_bar_addr == U64_MAX)
42684339 rc = -EIO;
42694340
4341+ } else if (addr >= HOST_PHYS_BASE && addr < host_phys_end &&
4342+ user_address && !iommu_present(&pci_bus_type)) {
4343+ *(u64 *) phys_to_virt(addr - HOST_PHYS_BASE) = val;
4344+
42704345 } else {
42714346 rc = -EFAULT;
42724347 }
@@ -4274,6 +4349,13 @@ static int goya_debugfs_write64(struct hl_device *hdev, u64 addr, u64 val) · 1 unmodified line
42744349 return rc;
42754350 }
42764351
4352+static int goya_debugfs_read_dma(struct hl_device *hdev, u64 addr, u32 size,
4353+ void *blob_addr)
4354+{
4355+ dev_err(hdev->dev, "Reading via DMA is unimplemented yet\n");
4356+ return -EPERM;
4357+}
4358+
42774359 static u64 goya_read_pte(struct hl_device *hdev, u64 addr)
42784360 {
42794361 struct goya_device *goya = hdev->asic_specific;
@@ -4401,6 +4483,8 @@ static const char *_goya_get_event_desc(u16 event_type) · 121 unmodified lines
44014483 return "THERMAL_ENV_S";
44024484 case GOYA_ASYNC_EVENT_ID_FIX_THERMAL_ENV_E:
44034485 return "THERMAL_ENV_E";
4486+ case GOYA_ASYNC_EVENT_PKT_QUEUE_OUT_SYNC:
4487+ return "QUEUE_OUT_OF_SYNC";
44044488 default:
44054489 return "N/A";
44064490 }
@@ -4483,6 +4567,9 @@ static void goya_get_event_desc(u16 event_type, char *desc, size_t size) · 76 unmodified lines
44834567 index = event_type - GOYA_ASYNC_EVENT_ID_DMA_BM_CH0;
44844568 snprintf(desc, size, _goya_get_event_desc(event_type), index);
44854569 break;
4570+ case GOYA_ASYNC_EVENT_PKT_QUEUE_OUT_SYNC:
4571+ snprintf(desc, size, _goya_get_event_desc(event_type));
4572+ break;
44864573 default:
44874574 snprintf(desc, size, _goya_get_event_desc(event_type));
44884575 break;
@@ -4534,6 +4621,15 @@ static void goya_print_mmu_error_info(struct hl_device *hdev) · 45 unmodified lines
45344621 }
45354622 }
45364623
4624+static void goya_print_out_of_sync_info(struct hl_device *hdev,
4625+ struct cpucp_pkt_sync_err *sync_err)
4626+{
4627+ struct hl_hw_queue *q = &hdev->kernel_queues[GOYA_QUEUE_ID_CPU_PQ];
4628+
4629+ dev_err(hdev->dev, "Out of sync with FW, FW: pi=%u, ci=%u, LKD: pi=%u, ci=%u\n",
4630+ sync_err->pi, sync_err->ci, q->pi, atomic_read(&q->ci));
4631+}
4632+
45374633 static void goya_print_irq_info(struct hl_device *hdev, u16 event_type,
45384634 bool razwi)
45394635 {
@@ -4698,7 +4794,7 @@ void goya_handle_eqe(struct hl_device *hdev, struct hl_eq_entry *eq_entry) · 158 unmodified lines
46984794 case GOYA_ASYNC_EVENT_ID_PSOC_GPIO_05_SW_RESET:
46994795 goya_print_irq_info(hdev, event_type, false);
47004796 if (hdev->hard_reset_on_fw_events)
4701− hl_device_reset(hdev, true, false);
4797+ hl_device_reset(hdev, HL_RESET_HARD);
47024798 break;
47034799
47044800 case GOYA_ASYNC_EVENT_ID_PCIE_DEC:
@@ -4754,6 +4850,15 @@ void goya_handle_eqe(struct hl_device *hdev, struct hl_eq_entry *eq_entry) · 49 unmodified lines
47544850 goya_unmask_irq(hdev, event_type);
47554851 break;
47564852
4853+ case GOYA_ASYNC_EVENT_PKT_QUEUE_OUT_SYNC:
4854+ goya_print_irq_info(hdev, event_type, false);
4855+ goya_print_out_of_sync_info(hdev, &eq_entry->pkt_sync_err);
4856+ if (hdev->hard_reset_on_fw_events)
4857+ hl_device_reset(hdev, HL_RESET_HARD);
4858+ else
4859+ hl_fw_unmask_irq(hdev, event_type);
4860+ break;
4861+
47574862 default:
47584863 dev_err(hdev->dev, "Received invalid H/W interrupt %d\n",
47594864 event_type);
@@ -5083,7 +5188,7 @@ static int goya_mmu_invalidate_cache(struct hl_device *hdev, bool is_hard, · 323 unmodified lines
50835188 if (rc) {
50845189 dev_err_ratelimited(hdev->dev,
50855190 "MMU cache invalidation timeout\n");
5086− hl_device_reset(hdev, true, false);
5191+ hl_device_reset(hdev, HL_RESET_HARD);
50875192 }
50885193
50895194 return rc;
@@ -5134,7 +5239,7 @@ static int goya_mmu_invalidate_cache_range(struct hl_device *hdev, · 44 unmodified lines
51345239 if (rc) {
51355240 dev_err_ratelimited(hdev->dev,
51365241 "MMU cache invalidation timeout\n");
5137− hl_device_reset(hdev, true, false);
5242+ hl_device_reset(hdev, HL_RESET_HARD);
51385243 }
51395244
51405245 return rc;
@@ -5160,7 +5265,7 @@ int goya_cpucp_info_get(struct hl_device *hdev) · 19 unmodified lines
51605265 if (!(goya->hw_cap_initialized & HW_CAP_CPU_Q))
51615266 return 0;
51625267
5163− rc = hl_fw_cpucp_info_get(hdev, mmCPU_BOOT_DEV_STS0, mmCPU_BOOT_ERR0);
5268+ rc = hl_fw_cpucp_handshake(hdev, mmCPU_BOOT_DEV_STS0, mmCPU_BOOT_ERR0);
51645269 if (rc)
51655270 return rc;
51665271
@@ -5443,6 +5548,7 @@ static const struct hl_asic_funcs goya_funcs = { · 276 unmodified lines
54435548 .debugfs_write32 = goya_debugfs_write32,
54445549 .debugfs_read64 = goya_debugfs_read64,
54455550 .debugfs_write64 = goya_debugfs_write64,
5551+ .debugfs_read_dma = goya_debugfs_read_dma,
54465552 .add_device_attr = goya_add_device_attr,
54475553 .handle_eqe = goya_handle_eqe,
54485554 .set_pll_profile = goya_set_pll_profile,
drivers/misc/habanalabs/goya/goyaP.h+2−0
@@ -49,6 +49,8 @@ · 48 unmodified lines
4949
5050 #define MAX_POWER_DEFAULT 200000 /* 200W */
5151
52+#define DC_POWER_DEFAULT 20000 /* 20W */
53+
5254 #define DRAM_PHYS_DEFAULT_SIZE 0x100000000ull /* 4GB */
5355
5456 #define GOYA_DEFAULT_CARD_NAME "HL1000"
drivers/misc/habanalabs/include/common/cpucp_if.h+98−1
@@ -11,6 +11,8 @@ · 10 unmodified lines
1111 #include <linux/types.h>
1212 #include <linux/if_ether.h>
1313
14+#include "hl_boot_if.h"
15+
1416 #define NUM_HBM_PSEUDO_CH 2
1517 #define NUM_HBM_CH_PER_DEV 8
1618 #define CPUCP_PKT_HBM_ECC_INFO_WR_PAR_SHIFT 0
@@ -28,6 +30,17 @@ · 11 unmodified lines
2830 #define CPUCP_PKT_HBM_ECC_INFO_HBM_CH_SHIFT 6
2931 #define CPUCP_PKT_HBM_ECC_INFO_HBM_CH_MASK 0x000007C0
3032
33+#define PLL_MAP_MAX_BITS 128
34+#define PLL_MAP_LEN (PLL_MAP_MAX_BITS / 8)
35+
36+/*
37+ * info of the pkt queue pointers in the first async occurrence
38+ */
39+struct cpucp_pkt_sync_err {
40+ __le32 pi;
41+ __le32 ci;
42+};
43+
3144 struct hl_eq_hbm_ecc_data {
3245 /* SERR counter */
3346 __le32 sec_cnt;
@@ -77,6 +90,7 @@ struct hl_eq_entry { · 43 unmodified lines
7790 struct hl_eq_ecc_data ecc_data;
7891 struct hl_eq_hbm_ecc_data hbm_ecc_data;
7992 struct hl_eq_sm_sei_data sm_sei_data;
93+ struct cpucp_pkt_sync_err pkt_sync_err;
8094 __le64 data[7];
8195 };
8296 };
@@ -287,6 +301,30 @@ enum pq_init_status { · 204 unmodified lines
287301 * The result is composed of 4 outputs, each is 16-bit
288302 * frequency in MHz.
289303 *
304+ * CPUCP_PACKET_POWER_GET
305+ * Fetch the present power consumption of the device (Current * Voltage).
306+ *
307+ * CPUCP_PACKET_NIC_PFC_SET -
308+ * Enable/Disable the NIC PFC feature. The packet's arguments specify the
309+ * NIC port, relevant lanes to configure and one bit indication for
310+ * enable/disable.
311+ *
312+ * CPUCP_PACKET_NIC_FAULT_GET -
313+ * Fetch the current indication for local/remote faults from the NIC MAC.
314+ * The result is 32-bit value of the relevant register.
315+ *
316+ * CPUCP_PACKET_NIC_LPBK_SET -
317+ * Enable/Disable the MAC loopback feature. The packet's arguments specify
318+ * the NIC port, relevant lanes to configure and one bit indication for
319+ * enable/disable.
320+ *
321+ * CPUCP_PACKET_NIC_MAC_INIT -
322+ * Configure the NIC MAC channels. The packet's arguments specify the
323+ * NIC port and the speed.
324+ *
325+ * CPUCP_PACKET_MSI_INFO_SET -
326+ * set the index number for each supported msi type going from
327+ * host to device
290328 */
291329
292330 enum cpucp_packet_id {
@@ -320,6 +358,13 @@ enum cpucp_packet_id { · 27 unmodified lines
320358 CPUCP_PACKET_PCIE_REPLAY_CNT_GET, /* internal */
321359 CPUCP_PACKET_TOTAL_ENERGY_GET, /* internal */
322360 CPUCP_PACKET_PLL_INFO_GET, /* internal */
361+ CPUCP_PACKET_NIC_STATUS, /* internal */
362+ CPUCP_PACKET_POWER_GET, /* internal */
363+ CPUCP_PACKET_NIC_PFC_SET, /* internal */
364+ CPUCP_PACKET_NIC_FAULT_GET, /* internal */
365+ CPUCP_PACKET_NIC_LPBK_SET, /* internal */
366+ CPUCP_PACKET_NIC_MAC_CFG, /* internal */
367+ CPUCP_PACKET_MSI_INFO_SET, /* internal */
323368 };
324369
325370 #define CPUCP_PACKET_FENCE_VAL 0xFE8CE7A5
@@ -391,6 +436,12 @@ struct cpucp_unmask_irq_arr_packet { · 65 unmodified lines
391436 __le32 irqs[0];
392437 };
393438
439+struct cpucp_array_data_packet {
440+ struct cpucp_packet cpucp_pkt;
441+ __le32 length;
442+ __le32 data[0];
443+};
444+
394445 enum cpucp_packet_rc {
395446 cpucp_packet_success,
396447 cpucp_packet_invalid,
@@ -459,6 +510,51 @@ enum cpucp_pll_type_attributes { · 62 unmodified lines
459510 cpucp_pll_pci,
460511 };
461512
513+/*
514+ * MSI type enumeration table for all ASICs and future SW versions.
515+ * For future ASIC-LKD compatibility, we can only add new enumerations.
516+ * at the end of the table (before CPUCP_NUM_OF_MSI_TYPES).
517+ * Changing the order of entries or removing entries is not allowed.
518+ */
519+enum cpucp_msi_type {
520+ CPUCP_EVENT_QUEUE_MSI_TYPE,
521+ CPUCP_NIC_PORT1_MSI_TYPE,
522+ CPUCP_NIC_PORT3_MSI_TYPE,
523+ CPUCP_NIC_PORT5_MSI_TYPE,
524+ CPUCP_NIC_PORT7_MSI_TYPE,
525+ CPUCP_NIC_PORT9_MSI_TYPE,
526+ CPUCP_NUM_OF_MSI_TYPES
527+};
528+
529+/*
530+ * PLL enumeration table used for all ASICs and future SW versions.
531+ * For future ASIC-LKD compatibility, we can only add new enumerations.
532+ * at the end of the table.
533+ * Changing the order of entries or removing entries is not allowed.
534+ */
535+enum pll_index {
536+ CPU_PLL = 0,
537+ PCI_PLL = 1,
538+ NIC_PLL = 2,
539+ DMA_PLL = 3,
540+ MESH_PLL = 4,
541+ MME_PLL = 5,
542+ TPC_PLL = 6,
543+ IF_PLL = 7,
544+ SRAM_PLL = 8,
545+ NS_PLL = 9,
546+ HBM_PLL = 10,
547+ MSS_PLL = 11,
548+ DDR_PLL = 12,
549+ VID_PLL = 13,
550+ BANK_PLL = 14,
551+ MMU_PLL = 15,
552+ IC_PLL = 16,
553+ MC_PLL = 17,
554+ EMMC_PLL = 18,
555+ PLL_MAX
556+};
557+
462558 /* Event Queue Packets */
463559
464560 struct eq_generic_event {
@@ -470,7 +566,6 @@ struct eq_generic_event { · 5 unmodified lines
470566 */
471567
472568 #define CARD_NAME_MAX_LEN 16
473−#define VERSION_MAX_LEN 128
474569 #define CPUCP_MAX_SENSORS 128
475570 #define CPUCP_MAX_NICS 128
476571 #define CPUCP_LANES_PER_NIC 4
@@ -533,6 +628,7 @@ struct cpucp_security_info { · 56 unmodified lines
533628 * @dram_size: available DRAM size.
534629 * @card_name: card name that will be displayed in HWMON subsystem on the host
535630 * @sec_info: security information
631+ * @pll_map: Bit map of supported PLLs for current ASIC version.
536632 */
537633 struct cpucp_info {
538634 struct cpucp_sensor sensors[CPUCP_MAX_SENSORS];
@@ -554,6 +650,7 @@ struct cpucp_info { · 15 unmodified lines
554650 __u8 pad[7];
555651 struct cpucp_security_info sec_info;
556652 __le32 reserved6;
653+ __u8 pll_map[PLL_MAP_LEN];
557654 };
558655
559656 struct cpucp_mac_addr {
drivers/misc/habanalabs/include/common/hl_boot_if.h+219−0
@@ -13,6 +13,8 @@ · 12 unmodified lines
1313
1414 #define BOOT_FIT_SRAM_OFFSET 0x200000
1515
16+#define VERSION_MAX_LEN 128
17+
1618 /*
1719 * CPU error bits in BOOT_ERROR registers
1820 *
@@ -73,6 +75,9 @@ · 54 unmodified lines
7375 * CPU_BOOT_ERR0_PLL_FAIL PLL settings failed, meaning that one
7476 * of the PLLs remains in REF_CLK
7577 *
78+ * CPU_BOOT_ERR0_DEVICE_UNUSABLE_FAIL Device is unusable and customer support
79+ * should be contacted.
80+ *
7681 * CPU_BOOT_ERR0_ENABLED Error registers enabled.
7782 * This is a main indication that the
7883 * running FW populates the error
@@ -92,6 +97,7 @@ · 13 unmodified lines
9297 #define CPU_BOOT_ERR0_PRI_IMG_VER_FAIL (1 << 10)
9398 #define CPU_BOOT_ERR0_SEC_IMG_VER_FAIL (1 << 11)
9499 #define CPU_BOOT_ERR0_PLL_FAIL (1 << 12)
100+#define CPU_BOOT_ERR0_DEVICE_UNUSABLE_FAIL (1 << 13)
95101 #define CPU_BOOT_ERR0_ENABLED (1 << 31)
96102
97103 /*
@@ -170,6 +176,20 @@ · 72 unmodified lines
170176 * is set to the PI counter.
171177 * Initialized in: linux
172178 *
179+ * CPU_BOOT_DEV_STS0_FW_LD_COM_EN Flexible FW loading communication
180+ * protocol is enabled.
181+ * Initialized in: preboot
182+ *
183+ * CPU_BOOT_DEV_STS0_FW_IATU_CONF_EN FW iATU configuration is enabled.
184+ * This bit if set, means the iATU has been
185+ * configured and is ready for use.
186+ * Initialized in: ppboot
187+ *
188+ * CPU_BOOT_DEV_STS0_DYN_PLL_EN Dynamic PLL configuration is enabled.
189+ * FW sends to host a bitmap of supported
190+ * PLLs.
191+ * Initialized in: linux
192+ *
173193 * CPU_BOOT_DEV_STS0_ENABLED Device status register enabled.
174194 * This is a main indication that the
175195 * running FW populates the device status
@@ -195,6 +215,9 @@ · 19 unmodified lines
195215 #define CPU_BOOT_DEV_STS0_CLK_GATE_EN (1 << 13)
196216 #define CPU_BOOT_DEV_STS0_HBM_ECC_EN (1 << 14)
197217 #define CPU_BOOT_DEV_STS0_PKT_PI_ACK_EN (1 << 15)
218+#define CPU_BOOT_DEV_STS0_FW_LD_COM_EN (1 << 16)
219+#define CPU_BOOT_DEV_STS0_FW_IATU_CONF_EN (1 << 17)
220+#define CPU_BOOT_DEV_STS0_DYN_PLL_EN (1 << 19)
198221 #define CPU_BOOT_DEV_STS0_ENABLED (1 << 31)
199222
200223 enum cpu_boot_status {
@@ -230,6 +253,7 @@ enum kmd_msg { · 29 unmodified lines
230253 KMD_MSG_SKIP_BMC,
231254 RESERVED,
232255 KMD_MSG_RST_DEV,
256+ KMD_MSG_LAST
233257 };
234258
235259 enum cpu_msg_status {
@@ -238,4 +262,199 @@ enum cpu_msg_status { · 2 unmodified lines
238262 CPU_MSG_ERR,
239263 };
240264
265+/* communication registers mapping - consider ABI when changing */
266+struct cpu_dyn_regs {
267+ uint32_t cpu_pq_base_addr_low;
268+ uint32_t cpu_pq_base_addr_high;
269+ uint32_t cpu_pq_length;
270+ uint32_t cpu_pq_init_status;
271+ uint32_t cpu_eq_base_addr_low;
272+ uint32_t cpu_eq_base_addr_high;
273+ uint32_t cpu_eq_length;
274+ uint32_t cpu_eq_ci;
275+ uint32_t cpu_cq_base_addr_low;
276+ uint32_t cpu_cq_base_addr_high;
277+ uint32_t cpu_cq_length;
278+ uint32_t cpu_pf_pq_pi;
279+ uint32_t cpu_boot_dev_sts0;
280+ uint32_t cpu_boot_dev_sts1;
281+ uint32_t cpu_boot_err0;
282+ uint32_t cpu_boot_err1;
283+ uint32_t cpu_boot_status;
284+ uint32_t fw_upd_sts;
285+ uint32_t fw_upd_cmd;
286+ uint32_t fw_upd_pending_sts;
287+ uint32_t fuse_ver_offset;
288+ uint32_t preboot_ver_offset;
289+ uint32_t uboot_ver_offset;
290+ uint32_t hw_state;
291+ uint32_t kmd_msg_to_cpu;
292+ uint32_t cpu_cmd_status_to_host;
293+ uint32_t reserved1[32]; /* reserve for future use */
294+};
295+
296+/* HCDM - Habana Communications Descriptor Magic */
297+#define HL_COMMS_DESC_MAGIC 0x4843444D
298+#define HL_COMMS_DESC_VER 1
299+
300+/* this is the comms descriptor header - meta data */
301+struct comms_desc_header {
302+ uint32_t magic; /* magic for validation */
303+ uint32_t crc32; /* CRC32 of the descriptor w/o header */
304+ uint16_t size; /* size of the descriptor w/o header */
305+ uint8_t version; /* descriptor version */
306+ uint8_t reserved[5]; /* pad to 64 bit */
307+};
308+
309+/* this is the main FW descriptor - consider ABI when changing */
310+struct lkd_fw_comms_desc {
311+ struct comms_desc_header header;
312+ struct cpu_dyn_regs cpu_dyn_regs;
313+ char fuse_ver[VERSION_MAX_LEN];
314+ char cur_fw_ver[VERSION_MAX_LEN];
315+ /* can be used for 1 more version w/o ABI change */
316+ char reserved0[VERSION_MAX_LEN];
317+ uint64_t img_addr; /* address for next FW component load */
318+};
319+
320+/*
321+ * LKD commands:
322+ *
323+ * COMMS_NOOP Used to clear the command register and no actual
324+ * command is send.
325+ *
326+ * COMMS_CLR_STS Clear status command - FW should clear the
327+ * status register. Used for synchronization
328+ * between the commands as part of the race free
329+ * protocol.
330+ *
331+ * COMMS_RST_STATE Reset the current communication state which is
332+ * kept by FW for proper responses.
333+ * Should be used in the beginning of the
334+ * communication cycle to clean any leftovers from
335+ * previous communication attempts.
336+ *
337+ * COMMS_PREP_DESC Prepare descriptor for setting up the
338+ * communication and other dynamic data:
339+ * struct lkd_fw_comms_desc.
340+ * This command has a parameter stating the next FW
341+ * component size, so the FW can actually prepare a
342+ * space for it and in the status response provide
343+ * the descriptor offset. The Offset of the next FW
344+ * data component is a part of the descriptor
345+ * structure.
346+ *
347+ * COMMS_DATA_RDY The FW data has been uploaded and is ready for
348+ * validation.
349+ *
350+ * COMMS_EXEC Execute the next FW component.
351+ *
352+ * COMMS_RST_DEV Reset the device.
353+ *
354+ * COMMS_GOTO_WFE Execute WFE command. Allowed only on non-secure
355+ * devices.
356+ *
357+ * COMMS_SKIP_BMC Perform actions required for BMC-less servers.
358+ * Do not wait for BMC response.
359+ *
360+ * COMMS_LOW_PLL_OPP Initialize PLLs for low OPP.
361+ */
362+enum comms_cmd {
363+ COMMS_NOOP = 0,
364+ COMMS_CLR_STS = 1,
365+ COMMS_RST_STATE = 2,
366+ COMMS_PREP_DESC = 3,
367+ COMMS_DATA_RDY = 4,
368+ COMMS_EXEC = 5,
369+ COMMS_RST_DEV = 6,
370+ COMMS_GOTO_WFE = 7,
371+ COMMS_SKIP_BMC = 8,
372+ COMMS_LOW_PLL_OPP = 9,
373+ COMMS_INVLD_LAST
374+};
375+
376+#define COMMS_COMMAND_SIZE_SHIFT 0
377+#define COMMS_COMMAND_SIZE_MASK 0x1FFFFFF
378+#define COMMS_COMMAND_CMD_SHIFT 27
379+#define COMMS_COMMAND_CMD_MASK 0xF8000000
380+
381+/*
382+ * LKD command to FW register structure
383+ * @size - FW component size
384+ * @cmd - command from enum comms_cmd
385+ */
386+struct comms_command {
387+ union { /* bit fields are only for FW use */
388+ struct {
389+ unsigned int size :25; /* 32MB max. */
390+ unsigned int reserved :2;
391+ enum comms_cmd cmd :5; /* 32 commands */
392+ };
393+ unsigned int val;
394+ };
395+};
396+
397+/*
398+ * FW status
399+ *
400+ * COMMS_STS_NOOP Used to clear the status register and no actual
401+ * status is provided.
402+ *
403+ * COMMS_STS_ACK Command has been received and recognized.
404+ *
405+ * COMMS_STS_OK Command execution has finished successfully.
406+ *
407+ * COMMS_STS_ERR Command execution was unsuccessful and resulted
408+ * in error.
409+ *
410+ * COMMS_STS_VALID_ERR FW validation has failed.
411+ *
412+ * COMMS_STS_TIMEOUT_ERR Command execution has timed out.
413+ */
414+enum comms_sts {
415+ COMMS_STS_NOOP = 0,
416+ COMMS_STS_ACK = 1,
417+ COMMS_STS_OK = 2,
418+ COMMS_STS_ERR = 3,
419+ COMMS_STS_VALID_ERR = 4,
420+ COMMS_STS_TIMEOUT_ERR = 5,
421+ COMMS_STS_INVLD_LAST
422+};
423+
424+/* RAM types for FW components loading - defines the base address */
425+enum comms_ram_types {
426+ COMMS_SRAM = 0,
427+ COMMS_DRAM = 1,
428+};
429+
430+#define COMMS_STATUS_OFFSET_SHIFT 0
431+#define COMMS_STATUS_OFFSET_MASK 0x03FFFFFF
432+#define COMMS_STATUS_OFFSET_ALIGN_SHIFT 2
433+#define COMMS_STATUS_RAM_TYPE_SHIFT 26
434+#define COMMS_STATUS_RAM_TYPE_MASK 0x0C000000
435+#define COMMS_STATUS_STATUS_SHIFT 28
436+#define COMMS_STATUS_STATUS_MASK 0xF0000000
437+
438+/*
439+ * FW status to LKD register structure
440+ * @offset - an offset from the base of the ram_type shifted right by
441+ * 2 bits (always aligned to 32 bits).
442+ * Allows a maximum addressable offset of 256MB from RAM base.
443+ * Example: for real offset in RAM of 0x800000 (8MB), the value
444+ * in offset field is (0x800000 >> 2) = 0x200000.
445+ * @ram_type - the RAM type that should be used for offset from
446+ * enum comms_ram_types
447+ * @status - status from enum comms_sts
448+ */
449+struct comms_status {
450+ union { /* bit fields are only for FW use */
451+ struct {
452+ unsigned int offset :26;
453+ unsigned int ram_type :2;
454+ enum comms_sts status :4; /* 16 statuses */
455+ };
456+ unsigned int val;
457+ };
458+};
459+
241460 #endif /* HL_BOOT_IF_H */
drivers/misc/habanalabs/include/gaudi/gaudi.h+1−1
@@ -38,7 +38,7 @@ · 37 unmodified lines
3838
3939 #define QMAN_PQ_ENTRY_SIZE 16 /* Bytes */
4040
41−#define MAX_ASID 1024
41+#define MAX_ASID 2
4242
4343 #define PROT_BITS_OFFS 0xF80
4444
drivers/misc/habanalabs/include/gaudi/gaudi_async_events.h+2−0
@@ -303,6 +303,8 @@ enum gaudi_async_event_id { · 302 unmodified lines
303303 GAUDI_EVENT_NIC3_QP1 = 619,
304304 GAUDI_EVENT_NIC4_QP0 = 620,
305305 GAUDI_EVENT_NIC4_QP1 = 621,
306+ GAUDI_EVENT_DEV_RESET_REQ = 646,
307+ GAUDI_EVENT_PKT_QUEUE_OUT_SYNC = 647,
306308 GAUDI_EVENT_FIX_POWER_ENV_S = 658,
307309 GAUDI_EVENT_FIX_POWER_ENV_E = 659,
308310 GAUDI_EVENT_FIX_THERMAL_ENV_S = 660,
drivers/misc/habanalabs/include/gaudi/gaudi_async_ids_map_extended.h+27−16
@@ -301,10 +301,10 @@ static struct gaudi_async_events_ids_map gaudi_irq_map_table[] = { · 300 unmodified lines
301301 { .fc_id = 274, .cpu_id = 128, .valid = 0, .name = "" },
302302 { .fc_id = 275, .cpu_id = 128, .valid = 0, .name = "" },
303303 { .fc_id = 276, .cpu_id = 128, .valid = 0, .name = "" },
304− { .fc_id = 277, .cpu_id = 129, .valid = 0, .name = "" },
305− { .fc_id = 278, .cpu_id = 129, .valid = 0, .name = "" },
306− { .fc_id = 279, .cpu_id = 129, .valid = 0, .name = "" },
307− { .fc_id = 280, .cpu_id = 129, .valid = 0, .name = "" },
304+ { .fc_id = 277, .cpu_id = 129, .valid = 1, .name = "DMA_IF_SEI_0" },
305+ { .fc_id = 278, .cpu_id = 129, .valid = 1, .name = "DMA_IF_SEI_1" },
306+ { .fc_id = 279, .cpu_id = 129, .valid = 1, .name = "DMA_IF_SEI_2" },
307+ { .fc_id = 280, .cpu_id = 129, .valid = 1, .name = "DMA_IF_SEI_3" },
308308 { .fc_id = 281, .cpu_id = 130, .valid = 0, .name = "" },
309309 { .fc_id = 282, .cpu_id = 131, .valid = 0, .name = "" },
310310 { .fc_id = 283, .cpu_id = 132, .valid = 0, .name = "" },
@@ -670,18 +670,29 @@ static struct gaudi_async_events_ids_map gaudi_irq_map_table[] = { · 359 unmodified lines
670670 { .fc_id = 643, .cpu_id = 492, .valid = 0, .name = "" },
671671 { .fc_id = 644, .cpu_id = 493, .valid = 0, .name = "" },
672672 { .fc_id = 645, .cpu_id = 494, .valid = 0, .name = "" },
673− { .fc_id = 646, .cpu_id = 495, .valid = 0, .name = "" },
674− { .fc_id = 647, .cpu_id = 496, .valid = 0, .name = "" },
675− { .fc_id = 648, .cpu_id = 497, .valid = 0, .name = "" },
676− { .fc_id = 649, .cpu_id = 498, .valid = 0, .name = "" },
677− { .fc_id = 650, .cpu_id = 499, .valid = 0, .name = "" },
678− { .fc_id = 651, .cpu_id = 500, .valid = 0, .name = "" },
679− { .fc_id = 652, .cpu_id = 501, .valid = 0, .name = "" },
680− { .fc_id = 653, .cpu_id = 502, .valid = 0, .name = "" },
681− { .fc_id = 654, .cpu_id = 503, .valid = 0, .name = "" },
682− { .fc_id = 655, .cpu_id = 504, .valid = 0, .name = "" },
683− { .fc_id = 656, .cpu_id = 505, .valid = 0, .name = "" },
684− { .fc_id = 657, .cpu_id = 506, .valid = 0, .name = "" },
673+ { .fc_id = 646, .cpu_id = 495, .valid = 1, .name = "DEV_RESET_REQ" },
674+ { .fc_id = 647, .cpu_id = 496, .valid = 1,
675+ .name = "PKT_QUEUE_OUT_SYNC" },
676+ { .fc_id = 648, .cpu_id = 497, .valid = 1,
677+ .name = "STATUS_NIC0_ENG0" },
678+ { .fc_id = 649, .cpu_id = 498, .valid = 1,
679+ .name = "STATUS_NIC0_ENG1" },
680+ { .fc_id = 650, .cpu_id = 499, .valid = 1,
681+ .name = "STATUS_NIC1_ENG0" },
682+ { .fc_id = 651, .cpu_id = 500, .valid = 1,
683+ .name = "STATUS_NIC1_ENG1" },
684+ { .fc_id = 652, .cpu_id = 501, .valid = 1,
685+ .name = "STATUS_NIC2_ENG0" },
686+ { .fc_id = 653, .cpu_id = 502, .valid = 1,
687+ .name = "STATUS_NIC2_ENG1" },
688+ { .fc_id = 654, .cpu_id = 503, .valid = 1,
689+ .name = "STATUS_NIC3_ENG0" },
690+ { .fc_id = 655, .cpu_id = 504, .valid = 1,
691+ .name = "STATUS_NIC3_ENG1" },
692+ { .fc_id = 656, .cpu_id = 505, .valid = 1,
693+ .name = "STATUS_NIC4_ENG0" },
694+ { .fc_id = 657, .cpu_id = 506, .valid = 1,
695+ .name = "STATUS_NIC4_ENG1" },
685696 { .fc_id = 658, .cpu_id = 507, .valid = 1, .name = "FIX_POWER_ENV_S" },
686697 { .fc_id = 659, .cpu_id = 508, .valid = 1, .name = "FIX_POWER_ENV_E" },
687698 { .fc_id = 660, .cpu_id = 509, .valid = 1,
drivers/misc/habanalabs/include/gaudi/gaudi_fw_if.h+0−14
@@ -20,20 +20,6 @@ · 19 unmodified lines
2020 #define UBOOT_FW_OFFSET 0x100000 /* 1MB in SRAM */
2121 #define LINUX_FW_OFFSET 0x800000 /* 8MB in HBM */
2222
23−enum gaudi_pll_index {
24− CPU_PLL = 0,
25− PCI_PLL,
26− SRAM_PLL,
27− HBM_PLL,
28− NIC_PLL,
29− DMA_PLL,
30− MESH_PLL,
31− MME_PLL,
32− TPC_PLL,
33− IF_PLL,
34− PLL_MAX
35−};
36−
3723 enum gaudi_nic_axi_error {
3824 RXB,
3925 RXE,
drivers/misc/habanalabs/include/goya/goya.h+1−1
@@ -30,7 +30,7 @@ · 29 unmodified lines
3030
3131 #define QMAN_PQ_ENTRY_SIZE 16 /* Bytes */
3232
33−#define MAX_ASID 1024
33+#define MAX_ASID 2
3434
3535 #define PROT_BITS_OFFS 0xF80
3636
drivers/misc/habanalabs/include/goya/goya_async_events.h+1−0
@@ -188,6 +188,7 @@ enum goya_async_event_id { · 187 unmodified lines
188188 GOYA_ASYNC_EVENT_ID_HALT_MACHINE = 485,
189189 GOYA_ASYNC_EVENT_ID_INTS_REGISTER = 486,
190190 GOYA_ASYNC_EVENT_ID_SOFT_RESET = 487,
191+ GOYA_ASYNC_EVENT_PKT_QUEUE_OUT_SYNC = 506,
191192 GOYA_ASYNC_EVENT_ID_FIX_POWER_ENV_S = 507,
192193 GOYA_ASYNC_EVENT_ID_FIX_POWER_ENV_E = 508,
193194 GOYA_ASYNC_EVENT_ID_FIX_THERMAL_ENV_S = 509,
drivers/misc/habanalabs/include/goya/goya_fw_if.h+0−11
@@ -15,17 +15,6 @@ · 14 unmodified lines
1515 #define UBOOT_FW_OFFSET 0x100000 /* 1MB in SRAM */
1616 #define LINUX_FW_OFFSET 0x800000 /* 8MB in DDR */
1717
18−enum goya_pll_index {
19− CPU_PLL = 0,
20− IC_PLL,
21− MC_PLL,
22− MME_PLL,
23− PCI_PLL,
24− EMMC_PLL,
25− TPC_PLL,
26− PLL_MAX
27−};
28−
2918 #define GOYA_PLL_FREQ_LOW 50000000 /* 50 MHz */
3019
3120 #endif /* GOYA_FW_IF_H */
include/uapi/misc/habanalabs.h+57−20
@@ -297,6 +297,7 @@ enum hl_device_status { · 296 unmodified lines
297297 #define HL_INFO_SYNC_MANAGER 14
298298 #define HL_INFO_TOTAL_ENERGY 15
299299 #define HL_INFO_PLL_FREQUENCY 16
300+#define HL_INFO_POWER 17
300301
301302 #define HL_INFO_VERSION_MAX_LEN 128
302303 #define HL_INFO_CARD_NAME_MAX_LEN 16
@@ -410,6 +411,14 @@ struct hl_pll_frequency_info { · 107 unmodified lines
410411 __u16 output[HL_PLL_NUM_OUTPUTS];
411412 };
412413
414+/**
415+ * struct hl_power_info - power information
416+ * @power: power consumption
417+ */
418+struct hl_power_info {
419+ __u64 power;
420+};
421+
413422 /**
414423 * struct hl_info_sync_manager - sync manager information
415424 * @first_available_sync_object: first available sob
@@ -621,6 +630,7 @@ struct hl_cs_chunk { · 205 unmodified lines
621630 #define HL_CS_FLAGS_STAGED_SUBMISSION 0x40
622631 #define HL_CS_FLAGS_STAGED_SUBMISSION_FIRST 0x80
623632 #define HL_CS_FLAGS_STAGED_SUBMISSION_LAST 0x100
633+#define HL_CS_FLAGS_CUSTOM_TIMEOUT 0x200
624634
625635 #define HL_CS_STATUS_SUCCESS 0
626636
@@ -634,17 +644,10 @@ struct hl_cs_in { · 7 unmodified lines
634644 /* holds address of array of hl_cs_chunk for execution phase */
635645 __u64 chunks_execute;
636646
637− union {
638− /* this holds address of array of hl_cs_chunk for store phase -
639− * Currently not in use
640− */
641− __u64 chunks_store;
642−
643− /* Sequence number of a staged submission CS
644− * valid only if HL_CS_FLAGS_STAGED_SUBMISSION is set
645− */
646− __u64 seq;
647− };
647+ /* Sequence number of a staged submission CS
648+ * valid only if HL_CS_FLAGS_STAGED_SUBMISSION is set
649+ */
650+ __u64 seq;
648651
649652 /* Number of chunks in restore phase array. Maximum number is
650653 * HL_MAX_JOBS_PER_CS
@@ -656,8 +659,10 @@ struct hl_cs_in { · 5 unmodified lines
656659 */
657660 __u32 num_chunks_execute;
658661
659− /* Number of chunks in restore phase array - Currently not in use */
660− __u32 num_chunks_store;
662+ /* timeout in seconds - valid only if HL_CS_FLAGS_CUSTOM_TIMEOUT
663+ * is set
664+ */
665+ __u32 timeout;
661666
662667 /* HL_CS_FLAGS_* */
663668 __u32 cs_flags;
@@ -682,14 +687,46 @@ union hl_cs_args { · 18 unmodified lines
682687 struct hl_cs_out out;
683688 };
684689
690+#define HL_WAIT_CS_FLAGS_INTERRUPT 0x2
691+#define HL_WAIT_CS_FLAGS_INTERRUPT_MASK 0xFFF00000
692+
685693 struct hl_wait_cs_in {
686− /* Command submission sequence number */
687− __u64 seq;
688− /* Absolute timeout to wait in microseconds */
689− __u64 timeout_us;
694+ union {
695+ struct {
696+ /* Command submission sequence number */
697+ __u64 seq;
698+ /* Absolute timeout to wait for command submission
699+ * in microseconds
700+ */
701+ __u64 timeout_us;
702+ };
703+
704+ struct {
705+ /* User address for completion comparison.
706+ * upon interrupt, driver will compare the value pointed
707+ * by this address with the supplied target value.
708+ * in order not to perform any comparison, set address
709+ * to all 1s.
710+ * Relevant only when HL_WAIT_CS_FLAGS_INTERRUPT is set
711+ */
712+ __u64 addr;
713+ /* Target value for completion comparison */
714+ __u32 target;
715+ /* Absolute timeout to wait for interrupt
716+ * in microseconds
717+ */
718+ __u32 interrupt_timeout_us;
719+ };
720+ };
721+
690722 /* Context ID - Currently not in use */
691723 __u32 ctx_id;
692− __u32 pad;
724+ /* HL_WAIT_CS_FLAGS_*
725+ * If HL_WAIT_CS_FLAGS_INTERRUPT is set, this field should include
726+ * interrupt id according to HL_WAIT_CS_FLAGS_INTERRUPT_MASK, in order
727+ * not to specify an interrupt id ,set mask to all 1s.
728+ */
729+ __u32 flags;
693730 };
694731
695732 #define HL_WAIT_CS_STATUS_COMPLETED 0
@@ -999,8 +1036,8 @@ struct hl_debug_args { · 303 unmodified lines
9991036 * Each JOB will be enqueued on a specific queue, according to the user's input.
10001037 * There can be more then one JOB per queue.
10011038 *
1002− * The CS IOCTL will receive three sets of JOBS. One set is for "restore" phase,
1003− * a second set is for "execution" phase and a third set is for "store" phase.
1039+ * The CS IOCTL will receive two sets of JOBS. One set is for "restore" phase
1040+ * and a second set is for "execution" phase.
10041041 * The JOBS on the "restore" phase are enqueued only after context-switch
10051042 * (or if its the first CS for this context). The user can also order the
10061043 * driver to run the "restore" phase explicitly