Mission Critical

PricewaterhouseCoopers Keywords Magazine Vol. 9

Data driven

Two PwC alumni explain how Big Data is making the leap from the intelligence community to the private sector.

Jim Reagan
Data analytics have almost countless uses, but aren’t useful to a customer unless you marry the data with the right application.
Big Data—with a capital B—involves sifting through terabytes upon petabytes of information to draw connections, identify patterns and find meaningful analysis. From academia to the tech sector, it’s being hailed as a game-changer, with potential to revolutionize healthcare, spot business trends, stay one step ahead of criminals and combatants, and even change the way we record and view world history. But how?
To find out, Keyword recently spoke with Jim Reagan, senior vice president and CFO of The SI Organization, Inc., and Drew Perez, Intelligence Solutions at HIGHFLEET—two PwC alumni who are navigating a world of data that few dive deep enough to explore.
“It’s difficult to imagine, or even put a limit on, the importance of Big Data, whether it’s in defense or public services or the private sector,” says Reagan. “Twenty years from now, we’ll look back and realize we’re doing things we never could have imagined, thanks to having more available data and more ways to put it to work.”
Big Data is hardly a new concept. Ever since the first satellites, radar sensors, mobile devices and recording instruments started collecting and transmitting information, people have recognized the value of voluminous data. But they’ve also struggled to figure out how to use it. “The challenge is not one of gathering data, which happens continuously,” says Perez. “The problem is understanding what it means.”
For the intelligence community, finding such clarity can make all the difference. In the world of counterterrorism, make-or-break decisions involve national security, so the more complete the view of the information, the better. Big Data analysis helps agencies dealing with classified information forecast and deploy resources, as well as make tactical and strategic decisions. Perez points to the OODA Loop decision-making cycle (Observe, Orient, Decide and Act), a staple framework of intelligence created by Korean War fighter pilot Colonel John Boyd and taught everywhere from the War College to noncommissioned officer courses. Those who move through the cycle quicker, observing and reacting to events more rapidly than their competition, eventually “get inside” the opponent’s decision cycle to gain an edge.
“You live in their future,” Perez says. “You’re deciding and acting and operationalizing your decisions as they’re just orienting themselves and observing the environment—and you have a significant strategic and tactical advantage.”
Homing in on the most relevant information right away makes that edge possible. So how do you shave off seconds and potentially pile on an advantage? By understanding the meaning behind data that’s coming at you from all angles—and fast. The same logic applies in the private sector, where methodologies to make sense of overwhelming amounts of available information are still in the early stages. “There’s an untapped market for the use of Big Data,” Reagan says. “We’ve been helping government clients manage Big Data for years. Now, we’re eager to make it more available.”

Applications abound

Jim Reagan has been finding meaning in numbers since his early career as a staff auditor with Coopers & Lybrand in the 1980s. He moved along the audit path, primarily working in the real estate and government contracting sectors in Washington, D.C., where he was on the first audit team to ever comb through the Smithsonian’s financial books. “That was very cool,” he says. “It involved reconstructing a lot of very, very old financial records since they’d never been audited before.” Lessons in teamwork at the core of entry-level audit positions in public accounting firms—along with an education in real estate development and the government services sector—prepared him well for his early career posts.
Reagan went on to become CFO of PAE, Inc., which works to support the Defense and State Departments in Afghanistan, Iraq and Central Africa. Drawn by a connection to PAE’s mission to provide logistical support for peacekeeping and global security worldwide, Reagan helped refinance the company’s debt while developing cost structures to succeed in the ultra-competitive world of government contracting.
Now as the CFO at The SI Organization, Inc., Reagan is helping to diversify the company’s business to appeal to state and local governments, as well as private-sector customers. With potential belt-tightening looming, the SI is looking to Big Data analytics around geospatial data—the kind streaming in from commercial and government satellites, high-flying aircraft, radar and sensors that inform things like GPS devices, weather forecasts, commercial shipping routes, crop forecasts and tax assessments. As the data is cleared for release into the private sector, applications abound. “Data analytics have almost countless uses,” Reagan says. “But it’s not useful to a customer unless you marry the data with the right application.”
The SI hosts and brokers huge databases of geospatial information that both federal and local governments can repackage and make available to third-party providers. Their muscle memory built upon more than 40 years of supporting classified customers is strong, Reagan says. And now the SI is looking to leverage that know-how to build bridges between private-sector data owners, application providers and their customers. “We’re a data-driven company—that’s our legacy, it’s who we are,” Reagan says. “Data is behind every decision we make.”
Currently, the SI is working on applications that help governments at every level organize and cull data for value-added uses in the private sector, particularly in healthcare. Data analytics and capabilities enable more proactive forecasting, prevention and detection of Medicare and Medicaid fraud. Healthcare providers are also seeking to develop ways to better use electronic health records to diagnose patients, research disease and allow patients to play a more active role in their care.
“It’s everything from seeing someone’s symptoms at the doctor’s office to making sure the patient gets the right treatment so they don’t end up in the emergency room the next week,” Reagan says.
As anyone who’s filled out forms in a waiting room can attest, there’s plenty of data to be collected. But information is not intelligence—far from it. The whole nature of intelligence is supporting critical decisionmaking; information, however much, is just one small piece of the puzzle.

Plunging into the Deep Web

A former intelligence officer, Perez trained special forces in tradecraft—“your classic spy stuff,” he quips. From 2000 to 2002, he was a senior enterprise architect at Diamond Management and Technology Consultants, which joined with PwC in 2010. Perez spent most of his time with Diamond in Europe, focused on strategic technology transfer agreements. His work there helped him develop and refine skills within enterprise architectures, particularly when his experience forced him to simultaneously adopt corporate and national cultures.
Perez began his private-sector career by co-founding the Lockheed Martin Center for Security Analysis, where he helped create training programs for intelligence analysis and software used by the CIA, NSA and Defense Intelligence Agency. At the request of the Department of Homeland Security, he developed a declassified version of the program for a range of private- sector clients. “Pick any vertical, and you find a problem with too much data in too many places,” Perez says. “The core issue remains sense-making.”
One key technology is accessing Deep Web data—the kind that search engines and web crawlers don’t find— to create applications that efficiently convert disparate, disorganized data into structured, searchable formats. These applications understand foreign languages, apply link analysis to recognize relationships and use analytics to visualize the data. They’re all tools that matured in the counterterrorism effort but have yet to be taken advantage of in the private sector, and their capabilities provide a distinct competitive advantage for businesses. The ability to monitor and almost instantaneously make decisions on markets and customer behavior or to assess strategic position related to the competition, targeted demographic or market, Perez says, is key.
This is why Perez often gets the same set of questions from potential clients once they learn about his capabilities. “They ask, ‘How come we’ve never heard of this before?’ or ‘Why is it you guys know how to do it better than we do?’” Perez says. “Well, because I did work that used to be classified, that’s why.”
In counterinsurgency, support from the local populace is the cornerstone of success. To help win over “hearts and minds,” intelligence focuses on people’s past and potential behavior, along with the patterns that decision- makers hope to both understand and eventually influence. Similarly, in business, the support of the market provides the cornerstone to profitability. Applying time-tested methodologies and related technology to identify behavior patterns for markets, Perez says, can lead to an enormous advantage.
Drew Perez
Pick any vertical, and you find a problem with too much data in too many places. The core issue remains sense-making.
“It’s a no-brainer when you’re dealing with the intelligence community, because they’ve been doing this for decades,” Perez says. “In the private sector, it’s a significant investment—and you’ve got to articulate and justify return on investment.”
For Perez, there’s no better way to do that than to provide a real-world example of Big Data’s power. Case in point: A pharmaceutical client turned to Perez to leverage the counterterrorism tools he’d developed to identify counterfeiters. Perez architected and implemented a solution to monitor, in near real time, worldwide e-commerce activity connected to the pharmaceutical company’s product and figure out the probability that it may involve fraud. It took two weeks to set up the software, designed to troll well below the headwaters of the Internet indexed by search engines and web crawlers and to make sense of it all. Once they flipped the switch, a collection of data that might have taken the company months to compile took just hours to capture.
“By the seventh hour, we had saturation,” Perez says. “We were monitoring the whole planet. Anytime somebody engaged in any kind of transaction that mentioned that company’s products and violated the rules on pricing, I knew.” The capability is meaningful given what can be gained with the ability to monitor comparative product performance and market behavior in real time.

Big rewards, big challenges

Reagan says some clients come to the SI with an understanding of Big Data’s potential, and some don’t. One thing they do know is that the investment in infrastructure for data storage space and bandwidth can be huge. “Some of these agencies have terabytes of data, and for them, it’s a storage headache and a cost,” he says. “We can help show them how they can manage data more effectively and how they can get some return on the investment by making the data available for resale.”
Perez says ROI will become more tangible as better solutions are developed to link data in disparate locales: It’s not a question of whether or not we can get data; it’s a matter of knowing where the right data is. The information that decision-makers need may already exist internally within a company’s data stores, but individual files can still be literally all over the map.
“It could be sitting on a SharePoint file in Dubai, and that little piece of information has to be associated with a large data warehouse that’s sitting in Buenos Aires,” Perez says, noting that intelligence analysts simply don’t have the time or resources to go through all the data. But Perez says moving data to the same location is not the answer in this day and age. Technology exists that connects information seemingly far out of reach. “You don’t have to physically touch a data point,” Perez says. “You can associate it through depth of logic.”
Of course, storing, managing and capitalizing on the seemingly unlimited potential of data all come with the challenge of protecting it. According to PwC’s 2013 Global State of Information Security Survey, many organizations fail to perform thorough assessments of factors that contribute to breach-related financial losses. In fact, just over 25% of respondents considered damage to brand and reputation when estimating the full impact of a breach, while the same survey found 61% of respondents would stop using a company’s products or services after a breach.
From advanced persistent threats linked to foreign governments to insider threats made by disgruntled or corrupted employees, economic espionage is a growing concern. The White House has responded by issuing an executive order calling for the creation of a framework to reduce risk to critical infrastructure and ease sharing of threat information with the private sector. But individual organizations must find their own balance: Strong “need-to-know” control mechanisms must be enforced yet somehow tempered to allow collaboration. In PwC’s survey, more than 80% of respondents said protecting customer and employee data is important. Still, the percentage of respondents who reported an accurate inventory of employee and customer data remains below 40%.
To address gaps like these, the SI provides customers with technical advice and consulting around protecting national assets from the threat of cyberattack. Large corporations, banks and power companies may be dealing with thousands of attempted attacks per day, and Reagan says the SI is equipped to weather the storm. The company employs a Threat Operations Center (TOC) in Laurel, Maryland, to monitor its financial accounting system, human resource management system, payroll and internal email. It also has a separate TOC that’s focused on its customers’ needs, constantly monitoring, detecting and fending off cyberattacks from the outside. “All of our systems that have touch points to the Internet are protected,” says Reagan, who notes the SI customers’ data is kept on closed systems.
“In our work we’ve found significant evidence of Theft of Trade Secrets (ToTS) in our monitoring efforts. If sensitive material is not categorized and properly labeled based on the impact of unauthorized disclosure or dissemination—and personnel are not trained in the culture of information security—then it is very difficult to protect sensitive information and intellectual property,” Perez says.
The concerns are real, and new regulations to protect privacy and address issues of consent, collection, use and misuse of data are sure to come. As more data becomes available for more applications, challenges will undoubtedly continue to present themselves. But Reagan and Perez agree: Big Data’s power can’t be underestimated. In defense and intelligence, it’s proved to be an indispensable tool. Now in the private sector, Big Data has the potential to be nothing less than transformational. “Once you turn this stuff on and implement it,” Perez says, “you rule the market.”

A Cheap Spying Tool With a High Creepy Factor

AUGUST 2, 2013, 4:58 PM
Brendan O’Connor is a security researcher. How easy would it be, he recently wondered, to monitor the movement of everyone on the street – not by a government intelligence agency, but by a private citizen with a few hundred dollars to spare?
Mr. O’Connor, 27, bought some plastic boxes and stuffed them with a $25, credit-card size Raspberry Pi Model A computer and a few over-the-counter sensors, including Wi-Fi adapters. He connected each of those boxes to a command and control system, and he built a data visualization system to monitor what the sensors picked up: all the wireless traffic emitted by every nearby wireless device, including smartphones.
Each box cost $57. He produced 10 of them, and then he turned them on – to spy on himself. He could pick up the Web sites he browsed when he connected to a public Wi-Fi – say at a cafe – and he scooped up the unique identifier connected to his phone and iPad. Gobs of information traveled over the Internet in the clear, meaning they were entirely unencrypted and simple to scoop up.
Even when he didn’t connect to a Wi-Fi network, his sensors could track his location through Wi-Fi “pings.” His iPhone pinged the iMessage server to check for new messages. When he logged on to an unsecured Wi-Fi, it revealed what operating system he was using on what kind of device, and whether he was using Dropbox or went on a dating site or browsed for shoes on an e-commerce site. One site might leak his e-mail address, another his photo.
“Actually it’s not hard,” he concluded. “It’s terrifyingly easy.”
Also creepy – which is why he called his contraption “creepyDOL.”
“It could be used for anything depending on how creepy you want to be,” he said.
You could spy on your ex-lover, by placing the sensor boxes near the places the person frequents, or your teenage child, or the residents of a particular neighborhood. You could keep tabs on people who gather at a certain house of worship or take part in a protest demonstration in a town square. Their phones and tablets, Mr. O’Connor argued, would surely leak some information about them – and certainly if they then connected to an unsecured Wi-Fi. The boxes are small enough to be tucked under a cafe table or dropped from a hobby drone. They can be scattered around a city and go unnoticed.
Mr. O’Connor says he did none of that – and for a reason. In addition to being a security researcher and founder of a consulting firm called Malice Afterthought, he is also a law student at the University of Wisconsin at Madison. He says he stuck to snooping on himself – and did not, deliberately, seek to scoop up anyone else’s data – because of a federal law called the Computer Fraud and Abuse Act.
Some of his fellow security researchers have been prosecuted under that law. One of them, Andrew Auernheimer, whose hacker alias is Weev, was sentenced to 41 months in prison for exploiting a security hole in the computer system of AT&T, which made e-mail addresses accessible for over 100,000 iPad owners; Mr. Aurnheimer is appealing the case.
“I haven’t done a full deployment of this because the United States government has made a practice of prosecuting security researchers,” he contends. “Everyone is terrified.”
He is presenting his findings at two security conferences in Las Vegas this week, including at a session for young people. It is a window into how cheap and easy it is to erect a surveillance apparatus.
“It eliminates the idea of ‘blending into a crowd,’” is how he put it. “If you have a wireless device (phone, iPad, etc.), even if you’re not connected to a network, CreepyDOL will see you, track your movements, and report home.”
Can individual consumers guard against such a prospect? Not really, he concluded. Applications leak more information than they should. And those who care about security and use things like VPN have to connect to their tunneling software after connecting to a Wi-Fi hub, meaning that at least for a few seconds, their Web traffic is known to anyone who cares to know, and VPN does nothing to mask your device identifier.
In addition, every Wi-Fi network that your cellphone has connected to in the past is also stored in the device, meaning that as you wander by every other network, you share details of the Wi-Fi networks you’ve connected to in the past. “These are fundamental design flaws in the way pretty much everything works,” he said.



Creating Intelligence from Big Data

Exploiting big data, including the 90% of data hidden in the deep web, provides new insight for law enforcement, business, government and research. Learn how to create actionable intelligence from big data in this white paper, including:
  • The value of big data
  • Who can benefit most from big data
  • How to create understanding from big data
Download this whitepaper to learn why big data matters and how to create actionable insight from the deep web.

Not Only Structured Query Language


Amazon Dynamo distributed key value stores (Cassandra, VoltDB, Riak, Redis)

Google Big Table (Hbase)

Document Oriented Database (MongoDB, CouchDB, MarkLogic)

Graph Database (Neo4j)
Native Big Data Connectors Source: jasperforge.org

CLEAR (Consolidated Lead Evaluation and Reporting)

CLEAR next generation version of the investigative tool, AutoTrackXP, which has over a decade of history in the public records market. CLEAR launched to the law enforcement market in 2008 as ChoicePoint CLEAR (Consolidated Lead Evaluation and Reporting).

MeDICi Data Intensive Computing Framework

PNNL created a world-leading research program in Data Intensive Computing.

DIC is characterized by problems where data is the primary challenge, whether it is the complexity, size, or rate of the data acquisition. As the number of emerging scientific and national security problems continues to grow, so do our advancements in software and hardware architectures, analytics and visualization. We invite you to explore how PNNL is accelerating the speed of scientific discovery, decision support and threat detection across multiple disciplines.

Starlight Visual Information System (VIS)

A brief video demonstration of Starlight's capabilities including examples of social network analysis (SNA) features and web reporting functionality:

Text and UAV Video Analysis

Big data: The next frontier for innovation, competition, and productivity

Report from McKinsey Global Institute
Download Full Report (.pdf)

The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future.
MGI studied big data in five domains—healthcare in the United States, the public sector in Europe, retail in the United States, and manufacturing and personal-location data globally. Big data can generate value in each. For example, a retailer using big data to the full could increase its operating margin by more than 60 percent. Harnessing big data in the public sector has enormous potential, too. If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. And users of services enabled by personal-location data could capture $600 billion in consumer surplus. The research offers seven key insights.

1. Data have swept into every industry and business function and are now an important factor of production, alongside labor and capital. We estimate that, by 2009, nearly all sectors in the US economy had at least an average of 200 terabytes of stored data (twice the size of US retailer Wal-Mart's data warehouse in 1999) per company with more than 1,000 employees.

2. There are five broad ways in which using big data can create value. First, big data can unlock significant value by making information transparent and usable at much higher frequency. Second, as organizations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days, and therefore expose variability and boost performance. Leading companies are using data collection and analysis to conduct controlled experiments to make better management decisions; others are using data for basic low-frequency forecasting to high-frequency nowcasting to adjust their business levers just in time. Third, big data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services. Fourth, sophisticated analytics can substantially improve decision-making. Finally, big data can be used to improve the development of the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance (preventive measures that take place before a failure occurs or is even noticed).

3. The use of big data will become a key basis of competition and growth for individual firms. From the standpoint of competitiveness and the potential capture of value, all companies need to take big data seriously. In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value from deep and up-to-real-time information. Indeed, we found early examples of such use of data in every sector we examined.

4. The use of big data will underpin new waves of productivity growth and consumer surplus. For example, we estimate that a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent. Big data offers considerable benefits to consumers as well as to companies and organizations. For instance, services enabled by personal-location data can allow consumers to capture $600 billion in economic surplus.

5. While the use of big data will matter across sectors, some sectors are set for greater gains. We compared the historical productivity of sectors in the United States with the potential of these sectors to capture value from big data (using an index that combines several quantitative metrics), and found that the opportunities and challenges vary from sector to sector. The computer and electronic products and information sectors, as well as finance and insurance, and government are poised to gain substantially from the use of big data.

6. There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.

7. Several issues will have to be addressed to capture the full potential of big data. Policies related to privacy, security, intellectual property, and even liability will need to be addressed in a big data world. Organizations need not only to put the right talent and technology in place but also structure workflows and incentives to optimize the use of big data. Access to data is critical—companies will increasingly need to integrate information from multiple data sources, often from third parties, and the incentives have to be in place to enable this.

Podcast Download Distilling value and driving productivity from mountains of data

MGI senior fellow Michael Chui discusses how the scale and scope of companies' access to data is changing the way they do business.

Assessment of men; selection of personnel for the Office of Strategic Services [by] the OSS Assessment Staff

"This volume is the account of how a number of psychologists and psychiatrists attempted to assess the merits of men and women recruited for the Office of Strategic Services. The undertaking is reported because it represents the first attempt in America to design and carry out selection procedures in conformity with so-called organismic (Gestalt) principles. As a novel experiment it might interest a wide range of readers, but more specifically we hope it will invite the attention of those who are concerned with the problem of predicting human behavior, especially if they are engaged in practicing and developing clinical psychology and psychiatry and in improving present methods of diagnosis, assessment, and selection. All told, 5,391 recruits were studied intensively over a three-day period at one station or over a one-day period at another. These were the two areas in the United States where the bulk of assessment was done. Of these the performances of 1,187 who went overseas were described and rated by their superior officers and associates in the theater. Some standard procedures, elementalistic in design, were included in our program, because the best of these instruments are especially efficient in picking out disqualifying defects of function and so in eliminating men who arc definitely inferior. Organismic methods, on the other hand, are to be recommended in addition whenever it is necessary to discriminate unusual talent, to measure ability in the range running from low average to high superior. The plan described in this book was devised to fit the special needs of the Office of Strategic Services, but it would not take much ingenuity to modify some of the techniques and to invent others of the same type to meet the requirements of other institutions. These methods were first used on a large scale by Simoneit, as described in Wehrpsychologie, and the German military psychologists, and after them by the British"--Introduction. (PsycINFO Database Record (c) 2006 APA, all rights reserved).

Turning Firefox to an Ethical Hacking Platform

Security-database.com list of useful security auditing extensions :

- Information gathering

Whois and geo-location
ShowIP : Show the IP address of the current page in the status bar. It also allows querying custom services by IP (right mouse button) and Hostname (left mouse button), like whois, netcraft.
Shazou : The product called Shazou (pronounced Shazoo it is Japanese for mapping) enables the user with one-click to map and geo-locate any website they are currently viewing.
HostIP.info Geolocation : Displays Geolocation information for a website using hostip.info data. Works with all versions of Firefox.
Active Whois : Starting Active Whois to get details about any Web site owner and its host server.
Bibirmer Toolbar : An all-in-one extension. But auditors need to play with the toolbox. It includes ( WhoIs, DNS Report, Geolocation , Traceroute , Ping ). Very useful for information gathering phase

Enumeration / fingerprinting
Header Spy: Shows HTTP headers on statusbar
Header Monitor : This is Firefox extension for display on statusbar panel any HTTP response header of top level document returned by a web server. Example: Server (by default), Content-Encoding, Content-Type, X-Powered-By and others.

Social engineering
People Search and Public Record: This Firefox extension is a handy menu tool for investigators, reporters, legal professionals, real estate agents, online researchers and anyone interested in doing their own basic people searches and public record lookups as well as background research.

Googling and spidering
Advanced dork : Gives quick access to Google’s Advanced Operators directly from the context menu. This could be used to scan for hidden files or narrow in a target anonymously (via the scroogle.org option) [Updated Definition. Thanks to CP author of Advanced Dork]
SpiderZilla : Spiderzilla is an easy-to-use website mirror utility, based on Httrack from www.httrack.com.
View Dependencies : View Dependencies adds a tab to the "page info" window, in which it lists all the files which were loaded to show the current page. (useful for a spidering technique)

- Security Assessment / Code auditing

JSView : The ’view page source’ menu item now opens files based on the behavior you choose in the jsview options. This allows you to open the source code of any web page in a new tab or in an external editor.
Cert Viewer Plus : Adds two options to the certificate viewer in Firefox or Thunderbird: an X.509 certificate can either be displayed in PEM format (Base64/RFC 1421, opens in a new window) or saved to a file (in PEM or DER format - and PKCS#7 provided that the respective patch has been applied - cf.
Firebug : Firebug integrates with Firefox to put a wealth of development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page
XML Developer Toolbar:allows XML Developer’s use of standard tools all from your browser.

Headers manipulation
HeaderMonitor : This is Firefox extension for display on statusbar panel any HTTP response header of top level document returned by a web server. Example: Server (by default), Content-Encoding, Content-Type, X-Powered-By and others.
RefControl : Control what gets sent as the HTTP Referer on a per-site basis.
User Agent Switcher :Adds a menu and a toolbar button to switch the user agent of the browser

Cookies manipulation
Add N Edit Cookies : Cookie Editor that allows you add and edit "session" and saved cookies.
CookieSwap : CookieSwap is an extension that enables you to maintain numerous sets or "profiles" of cookies that you can quickly swap between while browsing
httpOnly : Adds httpOnly cookie support to Firefox by encrypting cookies marked as httpOnly on the browser side
Allcookies : Dumps ALL cookies (including session cookies) to Firefox standard cookies.txt file

Security auditing
HackBar : This toolbar will help you in testing sql injections, XSS holes and site security. It is NOT a tool for executing standard exploits and it will NOT learn you how to hack a site. Its main purpose is to help a developer do security audits on his code.
Tamper Data : Use tamperdata to view and modify HTTP/HTTPS headers and post parameters.
Chickenfoot : Chickenfoot is a Firefox extension that puts a programming environment in the browser’s sidebar so you can write scripts to manipulate web pages and automate web browsing. In Chickenfoot, scripts are written in a superset of Javascript that includes special functions specific to web tasks.

- Proxy/web utilities

FoxyProxy : FoxyProxy is an advanced proxy management tool that completely replaces Firefox’s proxy configuration. It offers more features than SwitchProxy, ProxyButton, QuickProxy, xyzproxy, ProxyTex, etc
SwitchProxy: SwitchProxy lets you manage and switch between multiple proxy configurations quickly and easily. You can also use it as an anonymizer to protect your computer from prying eyes
POW (Plain Old WebServer) : The Plain Old Webserver uses Server-side Javascript (SJS) to run a server inside your browser. Use it to distribute files from your browser. It supports Server-side JS, GET, POST, uploads, Cookies, SQLite and AJAX. It has security features to password-protect your site. Users have created a wiki, chat room and search engine using SJS.

- Misc

Hacks for fun
Greasemonkey : Allows you to customize the way a webpage displays using small bits of JavaScript (scripts could be download here)

Fire Encrypter : FireEncrypter is an Firefox extension which gives you encryption/decryption and hashing functionalities right from your Firefox browser, mostly useful for developers or for education & fun.

Malware scanner
QArchive.org web files checker : allowing people to check web files for any malware (viruses, trojans, worms, adware, spyware and other unwanted things) inclusions.
Dr.Web anti-virus link checker : This plugin allows you to check any file you are about to download, any page you are about to visit
ClamWin Antivirus Glue for Firefox : This extension scans every downloaded file automatically with ClamWin.

Anti Spoof
refspoof : Easy to pretend to origin from a site by overriding the url referrer (in a http request). — it incorporates this feature by using the pseudo-protocol spoof:// .. thus it’s possible to store the information in a "hyperlink" - that can be used in any context .. like html pages or bookmarks

Additional Links:

Blackbuntu is Ubuntu base distro for Penetration Testing with GNOME Desktop Environment. It's currently being built using the Ubuntu 10.10.

The Metasploit® Framework is a free, open source penetration testing solution developed by the open source community & Rapid7.

The Social-Engineer Toolkit (SET) is specifically designed to perform advanced attacks against the human element.

The intelligence community gets social by Brian Fung, WSJ

Digital media is mostly about entertainment for some, while for others, the value lies in being able to spread messages to a large audience. But, as many news organizations are discovering, Web 2.0 technologies are as good for listening as they are for broadcasting. The notion of social media as a trend-monitoring tool is spreading — and now U.S. spy agencies are jumping on board.

Intelligence Advanced Research Projects Activity (IARPA), the intelligence community’s research arm, says it hopes to use data gathered from social media to predict political unrest and natural disasters. While the proposal may rankle privacy critics, it’s just the latest example of the way intelligence officials are turning to the social Web to collect policy-relevant information.

The CIA already monitors social networks manually. In 2010, agency analysts became aware of a YouTube account allegedly belonging to the propaganda service of North Korea. Pyongyang soon had other identities set up on Twitter and Facebook (the latter of which was abandoned). The CIA issued several reports later that year on the regime’s entry into social media, concluding that the new Web offensive was primarily aimed at influencing the population of South Korea, one of the world’s most digitally enabled societies. Both countries are engaged in a tenuous military truce and longstanding public relations war.

Even as it was watching North Korea’s evolving positions on social media, the CIA was conducting a study of the social media landscape in India (pdf). Beyond uncovering some fascinating details about the country’s Internet usage patterns, analysts discovered that many of India’s controversial separatist groups were taking advantage of social media tools to advocate their agendas.

Spy agencies’ growing interest in digital media is perhaps unsurprising given that it is an industry that trades in information.. But it also reflects broader, underlying trends in intelligence-gathering. Since the end of the Cold War, U.S. officials have embraced what are called “open sources” — non-classified information drawn from newspapers, radio broadcasts and other publicly accessible outlets. Open sources accounted for some 80 percent (pdf)of what the CIA knew about the Soviet Union’s downfall in the early 1990s, according to then-deputy director William Studeman. Sherman Kent, one of the agency’s first analysts, once estimated (pdf) that 80 percent of all U.S. intelligence needs could be met with open sources in peacetime.

The biggest victory for open-source proponents came in 2005, when the CIA launched a new center dedicated to gleaning intelligence from public information. The announcement signaled more of a rebranding than anything else — open source intelligence has always been a part of the mix to some degree — but the event finally lent recognition and credibility to a historically obscure tradition.

The open-source revolution has only accelerated with social media. Now, analysts can tap directly into millions of individual sources at the micro level, examining tweets, blog posts and videos for new information. They can also step back and survey entire social ecosystems, using vast amounts of metadata to identify significant patterns of behavior in the abstract. Or at the mid-range level, digital media can reveal important connections among small groups of users.

Whether government scrutiny of social media is problematic for civil society depends on your conception of public and private. But it raises other questions, too. What is the intelligence value of an individual tweet? How does the study of social media affect signal-to-noise ratios and, more importantly, how does it affect ways in which the intelligence community allocates its resources to adapt? Does social media change the meaning of open-source intelligence?

Harris DirectionFinding and Geo-Location Systems

Cellular Phone Interception

Stingray/KingFish vehicicular-borne analog and digital interrogation, Direction Finding (DF), SIGINT collection; AmberJack Phased Array DF Antenna; Harpoon amplifier; Tarpon Software; LoggerHead handheld device: survey, intercept, interrogate analog and digital cellular networks; Seahorse interrogation and direction finding system; Triggerfish Multichannel analog and digital cellular network monitor (Link to Source)

Prophet Low Level Voice Intercept

Pen-Link Wireline, Wireless, VoIP, 3G, IP collection