Microsoft supercharges Fabric with new data tools to accelerate enterprise AI workflows

Today, Microsoft kicked off its Ignite conference, talking about all things AI, including how it has assembled the largest AI agent ecosystem and will allow enterprises to build more such apps using any of the 1,800 large language models it has on offer.

The move — a significant departure from the long-standing reliance on OpenAI — promises enhanced flexibility to developers, but we all know AI is just ‘garbage in and garbage out’ without a solid data foundation. 

To this end, Microsoft also announced a series of updates for Fabric, its end-to-end SaaS data platform. According to Arun Ulag, the corporate VP of Azure data, the biggest development is the integration of transactional databases, which will transform Fabric into a truly unified, open data platform bringing all the necessary technologies together in one place to build next-gen AI applications, including advanced agents. 

Other notable capabilities, some of which are being previewed while others are generally available, touch on different aspects of how Fabric operates, including data connectivity, workload performance, scalability, security and governance. 

“We are relatively early in the AI journey… There are many more customers, business users, and developers that can take advantage of these technologies. And as they take advantage of these tools, we have to evolve them. We have to drive costs further down and make sure that we further accelerate business value. Ultimately, all of this should translate into higher GDP growth for countries and stronger business outcomes for customers,” Ulag said in an interview with VentureBeat.

Transactional database integration for fast-tracked AI development

Microsoft launched Fabric last year as a SaaS-based data and analytics platform to bring its innovations across the data stack in one place. The unified offering leveraged several tools the company built over the years, including SQL Server, Excel, Power BI and Azure Synapse, and provided teams with an end-to-end experience to connect, manage and analyze large structured and unstructured data assets.

At the core, Fabric is underpinned by an open lakehouse architecture called OneLake. It serves as a central, multi-cloud repository that supports various open data formats (Apache Parquet, Delta Lake and Iceberg) and the downstream analytical workloads. In the last few months, both Fabric and OneLake have received several improvements, including Real-Time Intelligence – which now becomes generally available – for analyzing streaming logs, IoT and telemetry as well as tools for migrating data from other data environments.

However, running analytical workloads to identify trends and patterns is just one piece of the puzzle. AI is the real deal today, and for that, the users need to go beyond aggregated, historical data. To help with this, Microsoft has announced Fabric Databases, which will see different transactional databases plug into OneLake, allowing users to access both live data from transactional systems (think individual purchase or login events) and bulk analytical data through one unified layer.

The company is starting with the integration of its own Azure SQL database and will follow up with other transactional databases including including Cosmos DB (its NoSQL document database behind ChatGPT), PostgreSQL, MongoDB and Cassandra. It hopes the move will save developers from complex database integrations and enable them to power next-gen AI apps, managing billions of interactions daily.

“Built-in vector search, RAG support, and Azure AI integration simplify AI app development, and your data is instantly available in OneLake for advanced analytics. Developers can even use Copilot in Fabric to translate natural language queries into SQL and get inline code completion alongside code fixes and explanations,” Ulag noted in a blog post today.

OneLake catalog, new AI features and more 

In addition to transactional databases, Fabric is getting a new OneLake catalog to make it easier for teams to explore, manage and govern their entire Fabric data estate, no matter where the information has come from, as well as several AI capabilities to accelerate workflows.  

The catalog, as Ulag wrote in the blog, carries two main tabs: Explore and Govern. The former is generally available and will help teams discover and manage their trusted data. Meanwhile, the Govern tab, aimed at providing data owners with valuable insights, tools and recommendations for governing their data, is in preview at this stage. These features will ensure that the teams are aware of what’s going on across the platform, without running into any surprises. 

On the AI front, Microsoft is now previewing AI functions in Fabric notebooks, providing a simplified API for common AI text enrichments like summarization, translation, sentiment analysis, and more. The company is also enhancing AI skills (preview), which allow users to build agents that can be pointed to query any data across multiple systems via natural language. Ulag wrote AI skills now have an improved conversational experience. Plus, they can now connect to semantic models and Eventhouse KQL databases, going beyond lakehouse and data warehouse tables, mirrored DB and shortcut data.

Among other notable updates, Microsoft announced the general availability of API for GraphQL to allow efficient querying of multiple data sources using the widely adopted GraphQL technology; support for new events and simplified dashboard sharing in Real-Time Intelligence; and preview of open mirroring, a feature that allows any application or data provider to write data changes directly into a mirrored database within Fabric. It also confirmed the general availability of Azure SQL DB mirroring and the preview of SQL managed instance mirroring.

Finally, Fabric users will also get workspace monitoring and surge protection in preview. The former will provide detailed diagnostic logs for troubleshooting performance issues, capacity performance and data downtime, while the latter will prevent background jobs from starting after a set threshold. 

Microsoft Ignite runs from November 19 to November 22, 2024