Hey Yousfi, thanks for the feedback! Yeah, I fully agree with that! :)

Generally for batch processing job, I would schedule a job in Databricks to do the processing just to save on cluster costs. That job would persist the results to a data source (such as Azure SQL or Cosmos DB for example) then as you say connect that to a data visualization tool.

Real-time results are another challenge altogether and Power BI Desktop as far as I know doesn’t support that yet.