#big-data--spark
#big-data--spark
[ follow ]
#ai #machine-learning #databricks #snowflake #data-integration #generative-ai #scala #data-governance #observability
Tech industry
fromTechzine Global
1 month agoHPE offers AI at every scale for Nvidia's Vera Rubin portfolio
HPE offers plug-and-play Nvidia-powered AI solutions designed to integrate seamlessly with existing infrastructure across three customer groups: AI model builders, service providers, and regulated organizations.
Business intelligence
fromInfoWorld
1 month agoSnowflake's new 'autonomous' AI layer aims to do the work, not just answer questions
Project SnowWork is Snowflake's autonomous AI layer that automates data analysis tasks like forecasting, churn analysis, and report generation without requiring data team intervention.
Miscellaneous
fromTechzine Global
2 months agoKlarrio uses open source expertise to build foundational data platforms
Klarrio builds compliant, scalable open-source data platforms and platform-engineering foundations, integrating and securing underlying infrastructure so customers can focus on analytics and data science.
#ai-automation
Artificial intelligence
fromTechzine Global
1 month agoSnowflake's Project SnowWork targets autonomous enterprise AI
Snowflake launches Project SnowWork, an autonomous AI interface that performs enterprise tasks like forecasts and reports without data team involvement, expanding from backend infrastructure to front-office productivity tool.
Artificial intelligence
fromTechzine Global
1 month agoSnowflake's Project SnowWork targets autonomous enterprise AI
Snowflake launches Project SnowWork, an autonomous AI interface that performs enterprise tasks like forecasts and reports without data team involvement, expanding from backend infrastructure to front-office productivity tool.
fromMedium
3 months agoHow I Fixed a Critical Spark Production Performance Issue (and Cut Runtime by 70%)
"The job didn't fail. It just... never finished." That was the worst part. No errors.No stack traces.Just a Spark job running forever in production - blocking downstream pipelines, delaying reports, and waking up-on-call engineers at 2 AM. This is the story of how I diagnosed a real Spark performance issue in production and fixed it drastically, not by adding more machines - but by understanding Spark properly.
fromTechzine Global
2 months agoDatabricks makes serverless Postgress service Lakebase available
Databricks today announced the general availability of Lakebase on AWS, a new database architecture that separates compute and storage. The managed serverless Postgres service is designed to help organizations build faster without worrying about infrastructure management. When databases link compute and storage, every query must use the same CPU and memory resources. This can cause a single heavy query to affect all other operations. By separating compute and storage, resources automatically scale with the actual load.
Software development
fromInfoWorld
2 months agoAI is changing the way we think about databases
Developers have spent the past decade trying to forget databases exist. Not literally, of course. We still store petabytes. But for the average developer, the database became an implementation detail; an essential but staid utility layer we worked hard not to think about. We abstracted it behind object-relational mappers (ORM). We wrapped it in APIs. We stuffed semi-structured objects into columns and told ourselves it was flexible.
Software development
Artificial intelligence
fromInfoQ
2 months agoAutonomous Big Data Optimization: Multi-Agent Reinforcement Learning to Achieve Self-Tuning Apache Spark
A Q-learning agent autonomously learns and generalizes optimal Spark configurations by discretizing dataset features and combining with Adaptive Query Execution for superior performance.
fromTechzine Global
2 months agoDatabricks shows how AI strengthens the SaaS model
The rise of generative AI is often seen as an existential threat to the SaaS model. Interfaces would disappear, software would fade away, and existing players would become irrelevant. However, new figures from Databricks paint a different picture. Rather than undermining SaaS, AI appears to be increasing its use. This week, Databricks reported a revenue run rate of $5.4 billion, a 65 percent year-on-year increase. More than a quarter of that now comes from AI-related products.
Artificial intelligence
fromInfoWorld
2 months agoDatabricks adds MemAlign to MLflow to cut cost and latency of LLM evaluation
By replacing repeated fine‑tuning with a dual‑memory system, MemAlign reduces the cost and instability of training LLM judges, offering faster adaptation to new domains and changing business policies. Databricks' Mosaic AI Research team has added a new framework, MemAlign, to MLflow, its managed machine learning and generative AI lifecycle development service. MemAlign is designed to help enterprises lower the cost and latency of training LLM-based judges, in turn making AI evaluation scalable and trustworthy enough for production deployments.
Artificial intelligence
[ Load more ]