- Secret CTO
- Posts
- Secret CTO Newsletter | AI Benchmark Hype? Why Meta’s Llama Results Are Raising Eyebrows
Secret CTO Newsletter | AI Benchmark Hype? Why Meta’s Llama Results Are Raising Eyebrows
Plus, Google’s Gemini Live is going mainstream—here’s why CTOs should care.
Sophie Bruxner-Randall
April 09, 2025 • Estimated Reading Time: 7 minutes
Welcome to Secret CTO, your go-to source for expert insights, strategies, and trends to empower your technology leadership.
The Big Picture 📸
AI PRACTICES
Meta's recent venture into AI benchmarks with its new Llama 4-based models, Scout and Maverick, has stirred the industry. Maverick gained attention by achieving an ELO score of 1417 on the LMArena leaderboard, securing the second spot just below Gemini 2.5 Pro and outscoring well-known models like GPT-4o. However, this high ranking was attributed to a specially configured version intended for benchmark optimization, revealing a tactic to differentiate Meta's models in a competitive AI landscape.
This move highlights a broader industry trend where AI developers seek unique methods to distinguish their offerings in an evolving field. Customisation for benchmark performance isn't novel, as seen in other tech sectors, yet it underscores their drive for distinction amidst the ubiquitous capabilities of large language models. As companies innovate to outpace competitors with attributes like energy efficiency and response speed, this illustrates the continued strategic push for product differentiation in a crowded market.
CYBERSECURITY INNOVATIONS
Breach and attack simulation (BAS) tools are transforming cybersecurity by automating continuous threat testing. Key players like AttackIQ and Mandiant Security Validation illustrate the market's growth, but the emphasis should be on selecting a BAS solution that aligns with your organisational goals rather than merely chasing advanced features. It is essential to understand your firm's unique drivers—be it financial constraints, operational needs, or compliance requirements—and use this understanding to contextualise technical metrics, ensuring that the selected tool provides strategic value.
The adoption of automation has facilitated constant security testing, eliminating the need for manual red team-blue team exercises. Market leaders such as SafeBreach, Verodin, and AttackIQ exemplify this shift—SafeBreach's continuous security validation and AttackIQ’s use of the MITRE ATT&CK framework highlight their strategic relevance. As the BAS market anticipates a compound annual growth rate of 33.4%, reaching nearly $35 billion by 2029, BAS solutions are becoming indispensable in modern cybersecurity strategies, evolving from mere reactive vulnerability assessments to sophisticated, proactive security validation systems.
Reach the CTOs shaping the future of technology. Get your brand in front of the decision-makers who matter.
Tech Pulse 📊
AI COMPANION UPDATE
Microsoft Copilot's latest update transforms it into a proactive AI companion by introducing memory, personalised user profiles, and expanded agentic capabilities, enhancing productivity and decision-making for executives. The integration with apps and services like OpenTable and Expedia allows for seamless action taking, relevant for CTOs aiming to leverage AI for operational efficiency. Microsoft's enhancements position Copilot as a vital tool in competitive product management, aligning with professional growth and market leadership objectives.
TECHNOLOGY UPGRADE
Google is set to extend its Gemini Live upgrade, incorporating Project Astra's real-time camera and screen access, across a wider range of Android devices, no longer limiting it to Google Pixel and Samsung Galaxy devices. This development, enhancing Gemini's ability to interpret real-world data via mobile cameras, represents a significant strategic expansion, albeit requiring a Gemini Advanced subscription costing $19.99/month for full functionality. This broad availability positions Google to dominate in mobile AI capabilities, offering CPOs a critical chance to leverage next-gen technology for market leadership.
AI CONTENT CREATION
Google's NotebookLM update enhances AI podcast creation by autonomously sourcing information from the web, appealing to professionals seeking efficient content generation. The tool streamlines the learning process but raises ethical concerns about using third-party material without explicit permission. Users can easily convert web-sourced content into various media formats.
APPLE INTELLIGENCE UPDATES
Apple's iOS 18, launched alongside the iPhone 16 series, integrates advanced Apple Intelligence, such as Genmoji and Image Playground, enhancing user experience on select models like the iPhone 15 Pro and iPhone 16 lineup. This update, combined with enhanced security measures and AI-driven features like ChatGPT integration into Siri, reflects Apple's commitment to leading the smartphone market through continuous innovation and strategic product management.
The CTO’s Agenda 🗓️
APRIL 15-16, NEW YORK, NY
Taking place April 15–16 in New York, the AI in Finance Summit will spotlight cutting-edge applications of artificial intelligence across the banking, financial services, and insurance (BFSI) sectors. The event brings together senior tech leaders and CTOs to explore high-impact use cases, from leveraging advanced machine learning models to combat fraud, to deploying NLP-powered chatbots that elevate customer engagement. With a sharp focus on ethical, privacy-conscious AI adoption, this summit offers strategic insights into scalable, secure innovation—making it a must-attend for CTOs steering the future of financial technology.
Feedback Console 💻
How did this week’s edition deploy? |
Heads up! To ensure you continue receiving our newsletters, please add [email protected] to your contact list!
A publication from Contentive’s Technology Media Division