In a revelation today that has bounded amongst the tech news media the NY Times reported that Snowden was able to effectively use a spiderbot to crawl and mirror mass amounts of classified content. Embarrassing given a SME grade IDS would flag that activity.
What I want to know is did he even bother to forge his User-Agent string to match whatever browser they were using? Did he rate limit the requests to make it look remotely human?
I just can’t even understand why some rudimentary code at the web server side didn’t catch him. If I was serving out X number of pages to a single user and that is well over the assigned threshold then surely the user could be flagged.
If small to medium IT enterprise can work out the kinks in these issues and run reasonably secure businesses knowing that threats may come from within as well as from an external source then why can’t the government? We are talking about the fucking NSA. They should know that the number one rule of tradecraft is to trust nobody, and if you have to trust someone – divulge as little as you can to complete your operation. Hell, in IT we call it the principle of “least privilege”. A piece of software – or in this case a human asset – should have only the minimum level of privileges it requires to perform its task and no more.
Snowden continues to embarrass the government, even from afar. One must wonder about his motives and allegiances.
Sure, I am glad that he revealed to the world that the United States – a country that spews through its propaganda that it is a freedom loving place – actually has secret courts where you could be detained indefinitely for who the hell knows what. But we must ponder what drove him to such action, as he has already claimed the whole program was premeditated. Is Snowden himself an intelligence asset of another sovereign nation or just a whistleblower? We have heard that Assange allegedly was used as an asset by ASIO, the Australian foreign intelligence agency and therefore we must question Snowden and draw our own conclusions.
Mike the Crypto Goat 
When even you are coming around to the view of Snowden as a foreign asset… others should consider it meaningful. The Snowden Op bore fruit.
I expect there is far more to this story than meets the eye. I guess people need to read between the lines before considering Snowden as being merely a lone wolf – a single disaffected employee.
The explanation that Snowden was able to obtain these documents en masse simply by using a robot to crawl the NSA’s intranet sounds fanciful. Of all organizations the NSA would have IDS systems in place to detect this kind of activity.
@ Mike,
As you note it appears there was little or nothing to stop Ed Snowden “walking the branches and gathering the fruit” all the fruit not just those that hung low.
This was a very basic security problem that should not have been able to happen at any level, but it did.
The NSA has two basic missions in life, the first and most important is,
To protect the communications of the US from all others.
The second and actually lesser task in importance is,
To read the communications of all others friend or foe.
The two people responsable for ensuring these two are Clapper and Alexander. In their keenness to steam open letters they forgot or deliberatly ignored their primary job which was to stop others steaming open US letters.
The reesult was they not only left the stable door wide open over night they actually set fire to the hay. The result was not just one horse getting out but all horses bolting and not stopping untill long after the full light of the riseing sun was on their backs and the insurance company called…
So I guess it’s not to surprising that some regard the gross negligence by Alexander and Clapper as the equivalent of an “insurance scam” or “Fund raiser”.
Likewise some cannot beleive such negligence was possible and thus cognative disonance makes them imbue Ed Snowden with “god like powers” in order to explain what he accomplished.
Untill I see solid evidence otherwise I’m going with “dereliction of duty” by Alexander and Clapper, as Ocham’s razzor indicates.
Thus I suspect any method of very many available to Ed Snowden would have worked. And if this is the case then ai suspect other than satisfiying curiosity knowing the method is not going to realy add much to the security side.
As always I appreciate your experience and insight. I guess we can’t categorically say that the Snowden “op” wasn’t orchestrated by the US (or a foreign power) either. That’s the problem with all this spook stuff. . . the mind tries to consider all of the available possibilities where in any other field you would default to the most logical and simple
I guess it is just hard to believe an organization that is meant to be on the cutting edge of security would allow a contractor the privileges, and secondarily would not notice someone crawling their intranet. So I think it is a case of gross negligence on the part of the NSA or a deliberate oversight to allow these documents to come to light (ie it was engineered).