Beth Noveck was a member of the transition team and spent the 120 days following the president’s Memorandum on Transparency and Open Government of January 21 working to ready this project, which she joined us this week to discuss (see the sidebar for the full conversation). She is now the deputy chief technology officer for open government in the Office of Science and Technology Policy.

The January memorandum, the first of the administration, outlined the three guiding principles of transparency, participation, and collaboration, she explains. “The reason we want to have transparency is to create more accountability in government.” And participation from citizens, according to Noveck, is not solely a matter of inclusion; it also ensures that those working in government are getting the best expertise so they can make decisions about health care reform, environmental sustainability—and be certain those decisions are based on the best possible data and science.

Addressing the importance of the effort for the scientific community, Noveck pointed to Data.gov, the new catalog of bulk technical information created as part of the initiative. “If we don’t make it easy to find that information,” she said, “it’s very difficult for the scientific community to do research on it, to analyze it, to assess the quality, and then, in turn, to hold government accountable.”

Noveck has first-hand experience using technology tools to crowdsource research that informs government decisions. She spearheaded the successful Peer-to-Patent pilot project that allows volunteer experts to assist with the prior art research for patent applications. Their suggestions get voted on by other participants and the top finds became part of the docket of materials sent to the overworked patent examiners who might not otherwise know about the valuable resources or preexisting intellectual property.

The lesson from the project for the current initiative is clear: “The intelligence and expertise that we need to make the best quality decisions is not all located in Washington. We don’t have a monopoly on all the good information that we need to make decisions,” she said.

The administration’s government transparency work has drawn some criticism for hosting discussions with stakeholders behind closed doors—though information on the content or results of those talks, as well as other ideas on transparency, is now available on OSTP’s “From the Inbox” page. “For far too long, too much of the way that we have made policy has happened behind closed doors, without adequate opportunities for participation,” Noveck said, “and also without adequate rationale and feedback at the back end when decisions are made that actually justify and explain why a certain decision was made.”

The first public phase of the transparency idea-generation process is an online brainstorming session at the Open Government Dialogue. The final day to submit and vote on ideas is tomorrow, May 28th, after which the second “discussion” phase begins on June 3. When the coordinators—the White House Chief Technology Officer in OSTP, the Office of Management and Budget, and the General Services Administration—have digested all the input from this multi-stage process, OMB will prepare a set of open policy directives for federal departments, which have already been asked to create more of what Noveck calls “open government innovations.” A gallery of such innovations already in place appears on the White House website.

But part of open governance, Noveck emphasizes, is decentralization, and transparency is not the sole responsibility of any one person or single office. “Everyone is in charge. We’re all responsible,” she said.

At launch, Data.gov offers a small, diverse collection of 47 sets of “raw” data. These include spreadsheets of clean air trends, KML files for Google maps of tornado touchdowns, and XML files of patent information, among a variety of other things. The design is clean, easy to navigate, and draws layouts that will be familiar to users of other recent adminstration sites like Recovery.gov and HealthReform.gov. The initital offering is small compared to the large variety of datasets already available directly from federal agencies. But as Marshall Kilpatrick points out at ReadWriteWeb, “the Data.gov site says it was selective about quality and standards when choosing what to include.”

In a January 21 memorandum, the President laid out principles for ensuring that government is transparent, participatory, and collaborative. The memorandum directed the Chief Technology Officer to coordinate recommendations for an Open Government Directive “that instructs executive departments and agencies to take specific actions implementing the principles” outlined—within 120 days. As John Wonderlich of the Sunlight Foundation points out, the directive itself isn’t due today, which marks the end of that window. Moreover, President Obama’s pick for White House CTO, Aneesh Chopra, only had his confirmation hearing earlier this week.

The administration has published a request for public comment on open government recommendations in the Federal Registrar. Submissions are due June 19, but as Alyia Sternstein reports, several questions about how these recommendations will get organized and implemented remain.

As well, Sunlight Labs has been waiting for the launch and unveiled their Apps for America 2 contest to encourage designers and developers to build new tools that capitalize on the data.

Here’s a screen grab of their mockups:

This is precisiely the kind of work I’ve argued that the nonprofit and advocacy sphere needs to be engaged in right now. For decades, a host of think tanks has analyzed public policies and offered menus of solutions for how government operations should run. But some of the key lessons underscored by web technologies are that the presentation of ideas is inextricably linked to their content, and that effective policymaking must go hand-in-hand with effective communication. One of the great promises of this administration is radical transparency for government operations and information, and so there’s every reason for the experts in the tech and think tank worlds to leverage their design and policy knowledge in support of open government data and successful communications.

To that end, the Labs staff makes another key suggestion: Data.gov needs editors who can write about the information available. The site, they argue, “should feature data, blog about data, and perhaps even link off to interesting things that other people are doing with the data that comes from Data.gov.” Moreover, that sort of feedback loop will encourage more policy that drives further transparency.

Open data is going to change the way people interact with government online, and the Obama administration’s launch of Recovery.gov could become a test bed for this. George Thomas, the chief architect at the General Services Administration who led the Recovery.gov build using open-source software, has a vision that is making the tech community smile. George and his team are looking into ways to open up the data for stimulus projects in just the way I’ve described. (Technical readers can check out his slide presentation from a public event last weekend that illustrates his plans.)

Projects like Recovery.gov will help this cause if done right, but for a while still, the bottleneck with my “blue sky” dream for improving government transparency won’t be the technology – it will be the data. It has to be available and structured using standard conventions, which just means formatting data in a way that machines can interpret it regardless of software platform.

To give you an idea of just how quickly these open source frameworks can be turned into powerful transparency tools, take a look at the Apps for Democracy contest in Washington, DC last year. In November, the city sponsored a sort of “Iron Chef” competition for web developers to see what they could do with all the data it was releasing to the public. The city had just finished a significant project to make much of the city’s data available for public use and wanted to know what types of information were really in demand so they could help prioritize where the city should invest in opening up. The contest format was a great way to do this because it brought dozens of developers together with no up-front cost to the city, and allowed them to award a financial prize just to those projects deemed “winners.”

Forty-seven awesome submissions later, Apps for Democracy was a huge success. Teams built all sorts of applications, from tools that can help you find a carpool or help you find a parking meter, to apps that send information about the crime level in your neighborhood to your iPhone. By the organizers’ estimate, the project may have generated upwards of a 4000 percent return on investment. As a participant in the process with our team at Development Seed, my company, I was able to personally experience how much power open source provided once the right data was open. From concept to completion in less than 72 hours, we built a site called D.C. Bikes which aggregates data from the city data other online sources to provide a resource to bike commuters and enthusiasts.

The city’s GIS shape files provide the map layer for bike lanes; bike theft data came from city police reports; bike shop locations are included from Google; and tweets about biking in D.C. are piped in from Twitter; finally, the site also shows bikes for sale in D.C. on Craig’s List. All our team had to do was think about the users we wanted to serve and put the pieces together. With a little original content sprinkled in, we were able to offer the cycling community a helpful new resource. Open source tools lowered the barrier to entry for building a site like this because they required no financial investment for proprietary software, and because so much innovation is being done on open source tools to get them ready to facilitate open data projects.

For sure, the site (and others like it) could have been even more useful if more data was available. For instance, if the police department made bike-related traffic accidents visible, we could have mapped them quickly to visualize problem areas along bike routes, and the community could figure out ways to press the city government to investigate and take action to improve public safety. Increasing the volume of available data will have a snowball effect. By opening up the city’s data and encouraging citizens to use it creatively for the public, the government is making DC safer, more accessible, and just plain cooler for its citizens.

To be clear, the point here is the power of data, and not applications. The White House and government agencies will never be able to devote the time or energy to building all the applications that citizens might want – nor should they try. There’s been a lot of talk about the president’s technology ambitions, but the biggest possible online technology win that the Obama administration could achieve would be opening up as much information and data as possible to empower the community to build anything it wants. The potential benefits of this was best put into words by Personal Democracy founder Andrew Raisej at TransparencyCamp this past weekend in DC: “One government can’t solve problems for 300 million people, but 300 million people can solve problems for one government.”

If data and information are open, available, and well-structured, the power of crowds that we’ve seen transform the news industry with the advent of blogging could extend into just about every corner of the web. The work ahead is to just get the data ready.

Ian Cairns is project manager and an online strategist for Development Seed, an online communications strategy shop based in Washington, D.C. For the past six years, Development Seed has specialized in working with large NGO’s, and they have been leaders in the Drupal open source project.

The government transparency movement is waiting for a deluge of public data from Congress and the Obama administration. Developers are ready with open-source software and protocols for structuring data on everything from lobbying disclosures to pending legislation to stimulus allocations. And once the data is free and flowing through RSS feeds, Application Programming Interfaces, and Twitter messages, others are poised to mash it up in visualizations, plot it on Google maps, and ferry it out for discussion with social networking tools.

But as the Washington Post reports today, the administration isn’t moving fast enough for the transparency community, and the reason isn’t technological; it’s operational. That is, the new media team at the White House, which is currently taking a lot of the heat on transparency issues, knows what technology it needs and how to use it to share info about what President Obama is doing and to solicit citizen input on policymaking. But there are operational issues built into the 20th-century rules that govern how executive offices can—and cannot—post information online.

The lesson that the first issue in transparency isn’t technology, but rather operations, isn’t trivial. How do you break down the red tape to allow discussion of unfinalized agency decisions on official blogs? How you dedicate staff and resources to maintain the quality and accuracy of complex technical data? How do you do build the concept of transparency into the DNA of a department when many have spent the last two terms operating behind an executive-mandated veil? These sort of questions have to come before a decision on what sort of markup language is right for the data or how a discussion board for regulations rules will thread comments.

Over the weekend, a few hundred developers, new media strategists, government information experts, and technology advocates converged on George Washington University for the first Transparency Camp, a BarCamp-style “unconference” designed to drive the discussion about how to get the federal government to open up and put its information online. In a discussion early Saturday, technology guru and O’Reilly Media founder Tim O’Reilly posed the fundamentals like this: “What are the most basic primitives for open government?” To put it another way, where do we start before we get into software specs? The difference between operations and technology will be crucial as advocacy groups continue to work with federal agencies as the latter open their data and the former start to analyze it.

The second issue that must inform effective transparency policy is that it must shape good public policy outcomes. Transparency is, in and of itself, an important principle in progressive governance, but the technologies that will enable radical transparency are not themselves policy solutions. Transparency has to work in the service of larger public policy goals: improving access to affordable health care, preventing environmental degradation, and reducing inequalities in the education system, to name a few.

The onus is on the advocacy groups to take federal data, analyze it, and create compelling arguments for public policy changes. That loop running from open government data to compelling projects that influence policy changes and back to more open data is what will drive future sectors of the government to open up and future advocacy groups to innovate.

So what do we need first? Many discussions at Transparency Camp boiled it down to this: structured data. Too many arms of the government lock their data in pdfs, text files, scanned images, or difficult-to-navigate databases. At another panel, O’Reilly recommended that the systems to start overhauling first are the ones that developers are already scraping and scouring for information, like the THOMAS database that houses information about Congressional legislation. Notoriously hard to query, the main page of the site currently has a link at the top directing visitors to the stimulus bill—because it’s that hard to find. “Follow the alpha geeks,” O’Reilly said. If they’re already hacking a site, then maybe it’s time to build them an API to give them access to the information inside.

After people have the data and the tools to access it, the next priority will be effective case examples borne of collaboration between advocacy groups and federal agencies opening up their processes and information. To be sure, some of these collaborations will fail, and that’s okay; smart groups will take the lessons learned and apply them to the next effort. But if the nonprofit community simply comes to the government demanding transparency, gets it, and there are no identifiable public policy reforms that benefit the people on main street, then there won’t be continued momentum on the issue to warrant future collaboration.

The potential of radical transparency for government data is immense, but at the same time, there’s an analog to the current state of genetics research: in both arenas, we have huge amounts of data with only a limited number of effective interpretations we use to make informed decisions.

It’s hard to communicate the energy and potential of this tech-savvy community—it’s also hard to predict just how transparency with change the way government and citizens interact. Yet as SP contributor Nancy Scola argued at TechPresident, it’s the end of transparency camp, but the beginning of what’s next.

Images: flickr.com/Avelino_Maestas and flickr.com/justgrimes