Big Data and sensor technology could help the city identify construction sites that violate noise regulations.
God bless the people who have the stamina to force government bureaucrat control freaks to provide access to data. Notice there is little mention of police and FBI, law enforcement in general providing information. No mention that for every report made by citizens another report is being collected on who did what, when. Big data has its downside as well.
New York City has been a Big Data pioneer for decades. In the early 1990s, the city launched the CompStat data-driven policing system, so that, in the words of former NYPD department chief Lou Anemone, officers could stop “just running around answering 911 calls” and start analyzing patterns to prevent crime. Thanks in part to CompStat, major crimes in the city have since fallen by 80 percent. During the Michael Bloomberg mayoral years, the city used data to pinpoint dangerous intersections and driving habits, cutting traffic deaths by nearly a third. Today, thanks to advances in data-storage capacity as well as the ubiquity of smartphones and broadband access, New York has an unprecedented number of facts to analyze and act upon, CompStat-style, across all areas of government—from building inspection to noise reduction. But [. . .] we still need old-fashioned political will.
Cities are in the middle of what Daniel Doctoroff, a Bloomberg-era deputy mayor, calls their fourth modern revolution. In the eighteenth century, cities got the first: the steam engine, which made possible the industrial age. In the nineteenth century, electricity gave cities the subways and elevators they needed to fit more people into small spaces. In the twentieth century, the automobile made it easier for residents and workers to leave dense urban areas. And now, in the twenty-first century, cities are getting the “networked revolution” of large-scale data collection, both human and automated, as well as continuous connectivity to transmit and store those data.
And with better transparency laws, much of this Big Data becomes open data: information that everyone can see and use. Consider how one area of data reporting and collection has exploded in just a decade: New Yorkers’ complaints, questions, and observations about their city. Last year, more than 30 million people called or went online to 311, the city’s information and complaint system. That’s nearly four times the city’s population, and four times the number of people who called in 2003,
[. . .]
One basic problem of even highly functional cities, Flowers notes, is “too big a body, too little blanket,” and the traditional answer to that problem, he says, is “let me hire more people.”
Instead, Flowers used data to help him perform a critical task: protecting New Yorkers from firetraps. “We were getting 20,000 complaints a year,” he says, many via 311, “about illegal [apartment] conversions”—that is, landlords or tenants subdividing their properties, creating dangerous conditions for poor tenants. But building inspectors, [. . .] regarded all buildings equally [. . .] Working with the fire department and buildings department to analyze historical information, Flowers’s team discovered which buildings were more likely to have critical violations resulting in destruction and death—and made it a priority to inspect those buildings. “What we basically did is reconstruct the inspection experience” to be risk-based, he says. Now, every building gets a risk grade, which inspectors use as guides. Fire deaths and injuries have never been so low.
Another success during the Bloomberg years: “end-to-end 911 analysis,” that is, integrating different city agencies’ systems to learn how long it takes for ambulances to respond to 911 calls and if response times could be improved.
But, Flowers notes, he always kept in mind that his job was to use data to pursue “the mission of [a government] agency,” not to protect the bureaucracy.
New York’s worst natural disaster in recent memory—Superstorm Sandy— [. . .] created an environment where the city recognized the value of data sharing,” says Hidalgo. The Bloomberg administration then made sure that every location in the city has a single, consistent address, now available to the public. “Interagency cooperation on address location” is “fundamental to a city as dense as New York,” Hidalgo says. As data quality and sharing improve, firefighters responding at a particular address could someday receive a warning that police have gone to the home multiple times for domestic-violence incidents, or see that the building’s landlord had violations at other buildings for dividing them into illegal and unsafe apartments—and that the landlord might have been doing the same thing with the burning building.
Half a decade ago, the state-run Metropolitan Transportation Authority started making data feeds available for the public’s use. [. . .] The MTA is exploring other applications; it held a “hackathon” in early March so that volunteers vying for $2,000 in prizes could crunch data to help the authority speed up Staten Island bus times.
Open data can be deployed not only to improve government but also to help business owners cope with the government agencies that regulate them. Aileen Gemma Smith, a local entrepreneur, saw that after Sandy, small-business owners wanted to know such things as when a certain street would reopen. She noted that this lack of information, though more acute after a disaster, was chronic even in normal times. Business owners showing up for work would be surprised that the city had closed their street for repairs—meaning lost customers and thus lost revenue. “I went to shopkeepers and said, ‘I’m building this for you, talk to me about what’s important,’ ” she says. The app she launched, called Mind My Business, crawls through hundreds of the city’s data sets to give subscribers practical information: “the MTA is closing the subway stop near your store this weekend,” or “the city is repairing the sidewalk that goes by your shop this week,” or “the previous owner of the restaurant space you own got fined four times for the following violations.” “Data aren’t just for privileged folks doing research,” Smith says. “Open data is how I help the local bodega guy, how I help the diner that’s been there for 25 years.” Mind My Business has 2,000 local subscribers.
Amateurs can mine Big Data to improve the quality of life in the city, too, even if they know nothing about software coding. Paul Vogel, a Prospect Heights resident, got a camera for his bike a few years back because he “had a couple of bad run-ins” with reckless car and truck drivers, “and I’m really bad at remembering license plates.” When he gets home after a ride, he sends pictures of taxis and other for-hire cars whose drivers have violated various laws to 311. “I was surprised that the 311 system for [taxi complaints] is so efficient . . . that I could get someone fined,” he says. Over a year or so, his self-described “hobby” has earned the city about $30,000 in revenue. More important, he may have saved some lives, by deterring drivers he got fined from parking in bike lanes or running red lights again. Vogel tweets his successes. “Putting it out on social media has raised awareness a little bit,” he says. Several other people have contacted him to let him know that they’ve done the same thing.
In 1974, the New York Public Interest Research Group, or NYPIRG, a good-government organization, helped prevail upon the state legislature and Governor Malcolm Wilson to pass the nation’s third freedom-of-information law. The FOIL law, as reporters know it, gave the press and public the right to obtain any government documents unless, among other restrictions, they violated private citizens’ confidentiality. A decade later, good-government groups persuaded the city council to establish the commission on public information and communications, which, in turn, published the city’s first public-data directory in 1993. New Yorkers could now access a list showing every publicly available data set that the city maintained, even if they couldn’t actually see the data without special software and expertise.
Public-information laws have never been perfect. Government agencies sometimes charge onerous fees for documents, and they often interpret their mandate to protect privacy too broadly, forcing petitioners to spend money and time suing for what they should get for free. But the laws gave a generation of civic researchers a valuable new tool. Steven Romalewski, director of CUNY’s mapping service and a long-time veteran of data collection who worked for NYPIRG for 22 years, says that he spent much of his time there “accessing data and putting it to use.”
In the 1990s, Romalewski used information he gleaned via the law to map toxic-waste sites on Long Island, showing the public: “Here’s how close they are to parks, here’s how close they are to schools.” [. . .] “Information transmission has dramatically changed” in recent years, though, says Romalewski, so that it’s become “almost trivial” to download a data set that once could have taken months to obtain.
During the Bloomberg years, civic-minded citizens got a legal update for the connectivity age: the 2012 open-data law, which requires the city to make certain data sets available on a free online portal. Today, city agencies offer 1,400 data sets to download and examine, with no tools needed besides spreadsheet software—now free, thanks to Google—and some patience. If you’re up late, upset about a barking dog, you can find out within an hour or so how many other people have had the same complaint over the last six years, and where they live—DIY data collection and analysis that were unfathomable even 15 years ago.
Those responsible for providing the data, however, often resist its dissemination. “The open-data law was pushed by the [city] council, not the mayor,” says John Kaehny, of the New York City Transparency Working Group. Though Bloomberg supported more open data, the problem persists in government. The issue isn’t cost—the city was already collecting all the data it now releases automatically to the public through its open-data portal—but accountability. “Data can equal embarrassment,” says Kaehny. More data can make more people aware that the city has a problem—whether with slow ambulance-response times or a rising street population.
Another drawback, from government’s perspective, is loss of control. Kaehny notes that city agencies have historically preferred to release their data selectively to “trusted users.” Open-data policies make such data “feeding” harder. If information is power, those who once enjoyed exclusive access but no longer do become less powerful.
The biggest danger in the Big Data and open-data worlds may be complacency. Just because the city releases 1,400 data sets doesn’t mean that we know everything that we need to know. We still need FOIL, as well as journalists and civic activists, to ask for data that the city hasn’t already collected or won’t give out because no journalist or researcher has asked for them. One thing hasn’t changed: information that is free and easy to get is often the only information that the government wants you to see.
At some point, though, Big Data and open data run into an old-fashioned problem: the city knows what the information suggests that it should do—but it won’t do it. For years now, residents of Battery Park City and the rest of the lower Hudson waterfront have suffered from incessant noise from tourist helicopters, says John Dellapontas of Stop the Chop, a residents’ advocacy group. “We have flight data that we got from FOIL,” he notes—referring to the freedom-of-information law (see box) by which his group has petitioned the city and state to reveal how many helicopter flights take off from a city-owned helipad. He could show “round trips, one every minute,” flying by residents’ apartment windows. Yet the company that runs helicopter tours used the lack of 311 data to say that residents didn’t mind the flights. “That is true, but meaningless,” Dellapontas says. “We started doing 311, [but] we’d get a form response saying that it’s perfectly legal, which it is.” Instead, Stop the Chop “circumvented the 311 system and had 5,000 members e-mail blast” local politicians. “We basically created a more effective 311 system.” Stop the Chop won the data battle but lost the war. The compromise that the mayor came up with reduced flights to every two minutes, from every one minute—and extended the tour-helicopter company’s lease.
Midtown residents have had similar difficulty in getting the city to rezone land so that developers can’t build 1,000-foot-plus condominium towers that cast large shadows across public spaces. The “sunshine task force” of Manhattan’s Community Board 5, the arm of local government that is supposed to be closest to citizens’ needs, has spent more than two years amassing and analyzing property-records data, zoning rules, and building permits, and working with other nonprofits to show the consequences of poorly conceived construction. The first problem that local residents have run into is that data released by the city are often difficult to understand and analyze. When sharing information on, say, the property-rights transfers necessary to build super-tall towers, the city doesn’t put out a simple weekly list of all such transfers. Residents looking to learn about such activity in their neighborhoods must crawl online through hundreds of individual property records to see what activity occurred at those properties. “The way the information is buried, it becomes unusable,” says Layla Law-Gisiko, chairperson of the task force.
Second, even when residents are able to do the difficult work of presenting incontrovertible, easy-to-understand data, it’s still easy for the city council and the mayor to ignore those data. [. . .] “It really makes no sense,” says Law-Gisiko. “It’s really putting our democratic system in jeopardy.”
On the plus side, data’s role in public decision making has become increasingly hard to ignore. Even in New York’s city council, which often makes headlines for greenlighting poorly considered ideas, at least some proposals go down to defeat when they’re not backed by sufficient information.
In some cases, however, government finds itself outgunned by private-sector firms that manipulate data to circumvent laws meant to protect New Yorkers’ quality of life. Take the example of Airbnb, which allows New Yorkers to turn their apartments into hotel rooms for paying guests. Airbnb’s most lucrative service allows New Yorkers to rent out entire empty apartments to strangers. This business is illegal under state and city law: New Yorkers do not want their apartment buildings to become hotels or youth hostels. Enforcing the law, though, is difficult. The city must rely on 311 callers to report illegal apartment rentals; then it must dispatch enforcers who wait, sometimes for hours, to spot illegal activity. The city expends taxpayer resources to find out something that Airbnb already knows, from its internal databases: which rentals are illegal and which are not. Before releasing select data to reporters in December 2015, Airbnb scrubbed its listing of at least 1,000 illegal rentals, in order to make the data look better to journalists. Airbnb knows where and when its “hosts” are breaking the law; the city does not.
[. . .]
[. . .]
And New Yorkers, of course, have urban problems that data won’t solve. On many subway lines now, riders can look at “countdown clocks” to see when the next train will arrive—but knowing that the Lexington Avenue line train is coming in two minutes doesn’t change the fact that not everyone waiting on the platform will fit onto that train. New Yorkers don’t need Big Data to tell them that the subways are too crowded. Without better, smarter investment in the assets that our second modern urban revolution—electricity—gave us, we can’t fully capitalize on Big Data’s potential.
The Fourth Urban Revolution
How Big Data is changing life in cities, for governments and citizens alike