On a Friday full of tech-land denials and government distancing and no real answers about how the NRA's sweeping spy program actually works, Palanatir — the "Mysterious Silicon Valley Company Helping the NSA Spy on Americans" — now insists that its own "Prism" system for database mining has nothing to do with the NSA's data-mining "PRISM" program, but that's not going to calm many privacy fears, either. "Palantir's Prism platform is completely unrelated to any US government program of the same name," the company wrote in a statement provided to The Atlantic Wire. "Prism is Palantir's name for a data integration technology used in the Palantir Metropolis platform (formerly branded as Palantir Finance). This software has been licensed to banks and hedge funds for quantitative analysis and research." It's true that Palantir Metropolis used to go by the name Palantir Finance, according to this Quora thread. And the link describing Palantir's Prism platform falls under the "Metropolis Dev" section. But the coincidence, as well as the company's strong ties to the CIA, have been hard to ignore.
Indeed, one of the many remaining questions from Thursday night's revelation that the NSA is spying on Americans through nine major Internet companies is how, exactly, the government got "direct access" to databases if tech companies vigorously deny they do just that. Even The Washington Post, which obtained the presentation that led to the disclosure of a second NSA program in as many days, has backtracked on its stance that the companies had begun to "participate knowingly."
If Apple and Facebook and others didn't willingly allow access to their servers, and didn't know their data was being mined, how did the NSA crack in?
Palantir, a CIA-funded startup, might be the key to answering that question, at least according to the Internet sleuthers of the day. Mostly because, yes, the company, which Bloomberg Businessweek's Ashlee Vance and Brad Stone described as "the darling of the intelligence and law enforcement communities" in 2011, also happens to have a program called PRISM, as both Talking Points Memo and Business Insider pointed out.
Not only do the NSA's PRISM program and Palantir's Prism program share a name, but Palantir seemingly markets pretty much the exact service the NSA would need to gain direct access to databases without a backdoor from tech companies. The startup's own Prism overview describes the product as "a software component that lets you quickly integrate external databases into Palantir," which sounds a lot like what the Post said the NSA needed to make PRISM work: "From inside a company's data stream the NSA is capable of pulling out anything it likes." One of the first "examples" Palantir gives of how to use its Prism system has to do with "Connecting to Databases."
Even if Prism has nothing to do with PRISM, Palantir might be able to help, despite the denial. Christopher Soghoian, a graduate fellow at the Center for Applied Cybersecurity in the School of Informatics and Computing at Indiana University, imagined Palantir had such capabilities back in 2011. "Using Palantir technology, the FBI can now instantly compile thorough dossiers on U.S. citizens, tying together surveillance video outside a drugstore with credit-card transactions, cell-phone call records, e-mails, airplane travel records, and Web search information," he told Businesweek. The government also has a policy that "prevents federal agencies from building their own software when they have access to commercial alternatives," according to a Wired article. So the government would almost have to outsource that kind of work to a company like Palantir.
There are, of course, many other ways the government could have accessed these databases. A Gawker tipster explained another scenario:
The NSA has collected the SSL root certificates from the various tech companies voluntarily.
Those certificates allow the NSA to decrypt internet traffic they collect through other means (e.g. a traffic splitter or, wait for it, prism) at a major US internet backbone.
They could have been siphoning that information from particular countries (identified by IP address) for years but, without the SSL keys of the various services, that data would have been useless.
Using the SSL keys they can decrypt data as it flows through in real-time.
This would match up with the statements by the tech companies, and would obviate the need for the NSA to make a copy of Facebook etc's data.
This is also the easiest, cheapest way to do this – and their Powerpoint slide says it only costs $20M annually.
The Atlantic's Alexis Madrigal also linked to these graphs explaining how the government might have dealt with the scale of the data by using "Map Reduce." Or the NSA could have been accessed the data with the help of Accumulo, the Google-esque open-source project that lets government intelligence operations store vast amounts of data in a single platform software.
But even if the NSA uses these other tools, at the very least Palantir is helping the government put together various sorts of information to make dossiers, as the Wired article explains:
Palantir helped to identify connections among key individuals and organisations. Officials reported that this kind of painstaking detective work -- reading reports, piecing together clues, drawing links between people -- would have taken months without technological assistance. With the help of Palantir, large amounts of data from disparate sources were analysed within days.
As Palantir founder Alexander Karp explained to Businessweek's Vance and Stone, the company's overall mission is "to protect the Shire" — yes, like Lord of the Rings — and, well, that sounds rather patriotic. Of course, any company would be patriotic if government contracts helped it triple its profits every year since 2008.