The Sequence Scope: Transformers Continue Setting New Records
Weekly newsletter that discusses impactful ML research papers, cool tech releases, the money in AI, and real-life implementations.
The Sequence Scope is a summary of the most important published research papers, released technology and startup news in the AI ecosystem in the last week. This compendium is part of TheSequence newsletter. Data scientists, scholars, and developers from Microsoft Research, Intel Corporation, Linux Foundation AI, Google, Lockheed Martin, Cardiff University, Mellon College of Science, Warsaw University of Technology, Universitat Politècnica de València and other companies and universities are already subscribed to TheSequence.
📝 Editorial: Transformers Continue Setting New Records
Transformers can be considered one of the most important milestones in the last decade of machine learning. Since Google’s famous paper: Attention is All You Need, the machine learning community has been in a frantic race to adapt this new type of architecture. Models like Google’s BERT, OpenAI’s GPT-3, Facebook RoBERTa and Microsoft’s Turing-NLG have been breaking records. Most of the progress in transformers has been centered in the natural language understanding (NLU) space, but that’s rapidly changing. New forms of transformer architecture are evolving in domains such as time-series and computer vision.
Just this week, AI labs from Microsoft Research and OpenAI published new transformer architectures that are reaching important milestones in different areas. Microsoft’s DeBERTa model was able to achieve state-of-the-art performance across several language tasks, beating famous benchmarks. OpenAI released two transformer-based models: DALL·E and CLIP, which almost magically connect language and computer vision tasks. Similarly, we are seeing implementations of transformer architectures incorporated into several mainstream deep learning frameworks. Certainly a trend to follow closely this year.
BTW, we discussed transformers in the VERY FIRST ISSUE that was sent as TheSequence Edge.
🔺🔻TheSequence Scope — our Sunday edition with the industry’s development overview — is free. To receive high-quality educational content every Tuesday and Thursday, please subscribe to TheSequence Edge 🔺🔻
🗓 Next week in TheSequence Edge:
Edge#53: the concept of Prophet; Facebook’s Prophet time-series algorithm; PyTorch Forecasting that enables deep learning models for time-series forecasting
Edge#54: Facebook’s ReBeL, an open-source bot that can master poker and other imperfect information games
Now, let’s review the most important developments in the AI industry this week.
🔎 ML Research
Microsoft Language Model Breaks SuperGlue Records
Microsoft Research published a paper outlining DeBERTa, a transformer-based model that surpasses the famous SuperGlue benchmark for natural language understanding tasks ->read more on Microsoft Research blog
Connecting Text and Images
OpenAI introduced two transformer models that can combine computer vision and language intelligence. DALL·E is able to generate images from text captions while CLIP can learn visual concepts using language supervision ->read more in these blog posts about DALL·E and CLIP respectively.
Polite Conversational Agents
Researchers from Uber and Carnegie Mellon University published a paper proposing a model for injecting social and polite languages into conversational agents ->read more in the original paper
🤖 Cool AI Tech Releases
LinkedIn FastIngest
LinkedIn open-sourced FastIngest, a data ingestion pipeline based on Apache Gobblin that powers their machine learning and analytics workloads ->read more on the LinkedIn engineering team blog
💸 Money in AI
- AI-driven fraud prevention and digital identity solutions startup Kount struck an agreement to be acquired by consumer credit reporting agency Equifax for $640 million. Linked by AI, Kount’s Identity Trust Global Network TM combines trust and fraud signals from 32 billion annual interactions to block fraud in real-time and to enable personalized customer experiences.
- 🦄 Cloud security automation platform Lacework raised a $525 million growth round, valuing the company at over $1 billion. Lacework uses AI to detect anomalous behavior and aims to close security gaps across DevOps and cloud environments by identifying threats targeting cloud servers, containers, and accounts.
- 🦄 Data-analytics startup Dremio raised $135 million in series D funding at a post-money valuation of $1 billion. Dremio taps machine learning to recommend datasets to users and adapt catalogs in response to changes in schema and execution. It also algorithmically caches and indexes metadata as needed, in real-time and on the fly.
- 🦄 AI-enhanced sales platform SalesLoft raised $100 million in funding at a $1.1 billion valuation. The company automates human interactions between buyers and sellers — from finding and following up on leads, to helping them sell with virtual coaching tools, and then assisting in the post-sales process.
- Enterprise-focused visitor management software provider iLobby raised $100 million in a funding round. The company develops smart sensors to utilize the data for analyzing the usage of physical space and getting insights into human interactions within it.
- AI-based robotic system for effective recycling AMP Robotics raised $55 million in a Series B funding round. The startup leverages a combination of AI algorithms and physical robots to orchestrate sorting, picking, and placing tasks. Computer vision is used to process millions of images to map complex material streams, while deep learning is applied to continuously improve the identification and categorization of different materials by all sorts of categories, contextualizing and storing data about each item it perceives. Hiring.
- Autonomous vehicle software company Oxbotica raised $47 million in a Series B round. Oxbotica has developed GAN-based simulation technology that’s capable of generating thousands of photorealistic images in a minute. It uses created synthetic images to teach its software, producing experiences and rehearsals which are not real but generated down to the raindrops on camera lenses.
- AI-powered sports recording solution Veo Technologies raised $24.5 million in a Series B round of funding. Veo is a portable and affordable solution that enables sports teams to record and analyze matches and training sessions, without the need for a camera operator.
