Scale AI to evaluate large language models for Pentagon

Editor’s note: This article was updated Feb. 20, 2024, to more accurately reflect Scale AI’s tasks.

The U.S. Department of Defense selected Scale AI to help it test and evaluate generative artificial intelligence for military applications.

The California-based company announced the work with the Chief Digital and AI Office on Feb. 20, the same day the CDAO was scheduled to kick off its conference in Washington, D.C., featuring discussions on the topic.

Generative AI is capable of generating text, images or other data outputs using algorithmic models in response to user prompts. Large language models are a type of generative AI that uses statistical relationships among text documents or other inputs, either by itself or through a supervised training process, to pump out essays, computer code, human-like conversation and more. Scale AI will produce benchmarks for such systems under the contract.

The Defense Department has expressed increasing interest in generative AI, but its uses remain debated. While a smart assistant or chatbot could efficiently find files, answer frequently asked questions or dig up contact information, such tools can also fuel disinformation campaigns, spoofing attempts and cyberattacks. The CDAO in August launched Task Force Lima to study and guide generative AI for national security purposes.

RELATED

Boeing expands drone exams to Lockheed C-5 with eye on broader fleet

"You just can't do this stuff in a simulated environment," Boeing's Scott Belanger said of the Autonomous Aircraft Inspection initiative.

By Colin Demarest

“Testing and evaluating generative AI will help the DOD understand the strengths and limitations of the technology, so it can be deployed responsibly,” Alexandr Wang, the founder and chief executive of Scale, said in a statement. “Scale is honored to partner with the DOD on this framework.”

The Tuesday announcement did not include a dollar value for the work.

Wang in July told Congress outdated data-retention and -management policies were hamstringing the Defense Department. What is needed, he said at the time, is “doubling down on some of these fast procurement methods and ensuring that we continue to innovate.”

“AI systems are only as good as the data that they are trained on,” he added.

Scale in 2022 won a nearly $250 million contract to provide federal agencies access to its technologies. The blanket purchasing agreement was issued by the Joint AI Center, which was subsumed by the CDAO.

More In AI & ML

Continuing resolution would slow military modernization, services warn

The Navy, for example, would have $26 billion in the wrong places, and would need Congress to approve $13 billion in formal reprogrammings.

Missile warning payload delay could push back 2025 launch plans

The late payload is being developed by RTX and will be integrated onto a Lockheed Martin satellite.

Space Futures Command could begin limited operations this year

Chief of Space Operations Gen. Chance Saltzman announced the creation of the command Feb. 12 as part of a slate of organizational changes.

Space Force’s fixed-price push includes some exceptions, Calvelli says

While the Space Force's acquisition executive stands by his drive for fixed-price contracts, he acknowledged some programs require a more nuanced approach.

Embraer sees strong demand for C-390 Millennium airlifter in Asia

Officials said they plan to take the C-390 plane on display at the Singapore Airshow to other countries in the region before returning to Brazil.