Data Visualization

It's not just UK homeowners that are considering downsizing, one of the country's biggest landowners is doing it too: the government. We look at how the state estate has changed since 2011Get the dataMore data journalism and data visualisations from the GuardianThe government has been criticised in the past for its inefficient use of the buildings it owns. From the plans for small businesses to take over the disused premises to suggestions that they be turned over for community use, there has been pressure from a number of quarters that the government could save space and money.It appears that they've responded to that pressure. Data released by the Cabinet Office today shows that the government's estate has shrunk 10% from 10.3m square metres in 2011 to 9.2m square metres in January of this year. The data unfortunately does not include information on who these properties were sold to (or, if they weren't owned, who they are no longer being rented or leased from) what they are now being used for or the amount of money spent/saved per building. It does however show that, taken together, 190 government buildings split between 31 different departments or organisations shrunk by 1,081,535 square metres between 2011 and 2013.That does not mean that every government building shrunk. In fact, 39 of those included in the data actually increased in size and 31 didn't change at all. The biggest changes were:HM Revenue and Customs shaved off a whopping 205,000 square metres from its estate - cutting it down to a mere 1.1 million square metresThe next biggest shrinkage was for the Ministry of Justice's National Offender Management Service (NOMS) which lost 117,000 square metresHer Majesty's Court Service by contrast witnessed the largest growth (42,000 square metres) closely followed by the British Transport Police Authority who acquired new premises 41,000 square metres in sizeThe State of the Estate report also looked at other features of the government estate. It found that 764 buildings were constructed after 1990 out of the 2,719 government building that had their age profiled. 182 of those buildings date back to pre-1900.The report also looks at how sustainable the various government buildings are and finds that most are performing well in reducing greenhouse gas emissions, water and waste. And just like responsible homebuyers, the report looks at the energy performance of new government acquisitions. Topping the list are the new Ministry of Justice buildings in Caernarfon and Llandrindod Wells.The largest vacant public buildings were to be found in the North West and the South West where 41,00 and 40,000 square metres respectively were not in use.Though there isn't a breakdown per building, the report does show which government departments and organisations have the largest annual running costs. At £604m per year, it's perhaps unsurprising that the department dealing with prisons tops the bill - the Ministry of Justice. Speaking about the changes, Francis Maude, Minister for the Cabinet Office said:We are pushing forward our ambitious programme to sell property we no longer use and get out of expensive rentals.Since the last general election we have saved the taxpayer over a £1 billion and made space available in desirable central locations for redevelopment. This is not only about changing the way we work but generating an economic boost. The changes over the past two years are part of the Government Property Unit strategy, the next stage of which concentrates on putting the existing space to better use. What do you think about these changes? Do they reflect a sensible rationalisation of otherwise inefficient resources? Or is downsizing an inevitable consequence of public sector job cuts? Share your views via Twitter @MonaChalabi or @GuardianData or by posting a comment below.Download the data• DATA: download the full spreadsheetCan you do something with this data?• Flickr Please post your visualisations and mash-ups on our Flick

Reader Jack S. sent over this chart (link): The first problem readers encounter with this image is "What is MMI?"  I like to think of any presentation as a set of tearout pages. Even if the image is part of a book, or part of a deck of slides, once it is published, the writer should expect readers from tearing a sheet out and passing it along. In fact, you'd love to have people pass along your work. This means that when creating a plot such as this, the designer must explain what MMI is in the footnote. Yes, on every chart even if every chart in the report deals with MMI. MMI, I'm told, is some kind of metric of health care cost. *** What a mess. They are trying to use the metaphor of "measuring one's temperature", which I suppose is cute because MMI measures health care costs. Next, the designer chose to plot the index against the national average as opposed to the dollar amount of MMI. This presents a challenge since the thermometer does not have a natural baseline number. This is especially true on the Fahrenheit scale used in the U.S. Then, a map is introduced to place the major cities. The bulb of each thermometer now doubles as a dot on the map. This step is mind-boggling because the city labels aren't even on the map. So if you know where these cities are, you don't need the map for guidance but if you don't know the locations, you're as hopeless as before. How the data now gets onto the complex picture requires some deconstruction. First, start with a bar chart of the relative index (the third column of the table shown above). Then, chop off the parts below 85 (colored gray). Next, identify the cities that are below the national average (i.e. index You can see this by focusing only on the chart above the map. In other words, this part: To get from here to the version published, add a guiding line from each bar to the dot on the map for the corresponding city. Notice that a constant-length portion of each bar has been chopped off, and now each bar is augmented by some additional length that varies with the distance of the bar chart from the geographical location of the city as shown on the map below. For instance, Miami, which is furthest south, has the biggest distortion. *** The choice of 85 as a cutoff is arbitrary and inexplicable. If we really want to create a "cutoff" of sorts, we can use 100, which represents the national average. By plotting the gap between the city index and the national index, effectively, the percent difference, we also can use the sign of the difference to indicate above/below the national average, thus saving a color. *** One of the most telling signs of a failed chart is the appearance of the entire data set next to the chart. That's the essence of the self-sufficiency test.

Inflation in the UK has fallen for the first time since autumn 2012, hitting 2.4% in April. Get the full data over time, see how it compares with pay - and understand the new measures• Interactive inflation data explorer from Timetric• Get the data• More data journalism and data visualisations from the GuardianHow pay has fallen behind inflationUK inflation has fallen for the first time since autumn 2012, hitting 2.4% in April, the Office for National Statistics (ONS) revealed today. The ONS state that "this is the first time the 12-month rate has fallen since the increase in tuition fees contributed to the growth in the CPI rising to 2.7% in October 2012. The intervening six months have seen inflation remain relatively stable". Katie Allen writes today: Falling petrol prices pushed down UK inflation more than expected last month, bringing some relief to both households and Bank of England policymakers.The consumer price measure of inflation fell for the first time since last autumn and stood at 2.4% in April, down from 2.8% in March. The drop was driven by lower fuel costs and airfares as the oil price fell, more than offsetting a rise in food prices after damage to crops over the long winter, the Office for National Statistics (ONS) said.There is more than one method of measuring inflation. Up until today, we were mostly concerned with the consumer price index (CPI) measure - which is the government's favoured one. In September 2011, when the CPI stood at 5.2%, it had never been higher in recorded history. The CPI is important because it is used for uprating pensions, wages and benefits.The second measure is the retail price index (RPI) which has now been dumped as a national statistic (although the ONS will keep publishing it for a while).Instead we now have CPIH, which includes owner occupiers' housing costs and RPIJ - the new version of the Retail Prices Index (RPI). If you're interested, the old RPI measure stands at 2.9% this month, down from 3.3% last month.This is how the indexes compare:We have also added in pay data - and you can see how inflation has been racing ahead of average earnings. Even with the falls in previous months, people's wages are not increasing as fast as the cost of living.There are some important differences between the two ways the ONS measures inflation. The government prefers the consumer price index, which also includes services, housing, electricity, food, and transportation, but the retail price index covers more items. The RPI includes housing costs and is used for many pay negotiations and used to be used for pension payments. We've included both here - just click on the links on the spreadsheet. You can get the full list of items in the inflation basket here.If you want to see how different elements of inflation have changed, check out this interactive data explorer from Timetric.We have gathered all the data for inflation since June 1948. Let us know what you can do with this data.To find out more about how RPI and CPI differ, explore this set of interactive charts. The graphics show a breakdown of the different measures and components of inflation, and you can compare rates between countries.Download the dataDATA: UK inflation since the 1940s - CPI and RPIINTERACTIVE: how we visualised the dataMore dataMore data journalism and data visualisations from the GuardianWorld government data• Search the world's government data with our gatewayDevelopment and aid data• Search the world's global development data with our gatewayCan you do something with this data?• Flickr Please post your visualisations and mash-ups on our Flickr group• Contact us at data@guardian.co.uk• Get the A-Z of data• More at the Datastore directory• Follow us on Twitter• Like us on FacebookInflationEconomicsQuantitative easingFinancial crisisDeflationEconomic policyBank of EnglandGovernment dataOffice for National StatisticsGeorge OsborneMervyn KingAmi SedghiSimon Rogersguardian.co.uk © 2013 Guardian News and Media

What do emptier waters look like? This web aquarium shows declining fish populations over the past 100 years - and it uses more than 200 datasets to do it. We recommend that you closely analyse the data showing that the population of big fish has been decimated while small fish are now overly abundant. Or else you can leave it on your screen as a digital pet and a grim modern Tamagotchi.Mona Chalabi

In collaboration between USGS, NASA and TIME, Google released a quarter century of satellite imagery to see how the world has changed over time. The images were collected as part of an ongoing joint mission between the USGS and NASA called Landsat. Their satellites have been observing earth from space since the 1970s—with all of the images sent back to Earth and archived on USGS tape drives that look something like this example (courtesy of the USGS). We started working with the USGS in 2009 to make this historic archive of earth imagery available online. Using Google Earth Engine technology, we sifted through 2,068,467 images—a total of 909 terabytes of data—to find the highest-quality pixels (e.g., those without clouds), for every year since 1984 and for every spot on Earth. We then compiled these into enormous planetary images, 1.78 terapixels each, one for each year. Be sure to check out the Timelapse feature on Time.

Heterosexual civil partnerships are on the cards for the UK, thanks to a tabled amendment to government plans on gay marriage. Is there any evidence as to how many people want them?Read the reality check from Shiv Malik hereGet the dataMore data journalism from the GuardianWhen your stance on a subject risks you being branded as homophobic, disrespectful of religion, out of touch or anti-establishment it might be better to argue your case on the basis of cost. And with a subject as sensitive as gay marriage on the table, that's exactly the strategy many MPs appear to be adopting. Or at least indirectly. In fact, the current debate between MPs is about the potential price tag if heterosexual couples were also to be granted civil partnership rights but many suggest that, if passed, the amendment could derail the gay marriage bill.The estimated costs for amending the bill have varied between £90m to £4bn - leading critics such as the Conservative MP Tim Loughton to brand the numbers as "back of fag packet scaremongering". While Shiv Malik is taking a look at how the numbers stack up on the reality check blog, we're considering the number of people the legislation is likely to affect.Peter Tatchell, sponsor of Equal Love, a group campaigning in favour of both same-sex civil marriage and opposite-sex civil partnerships, said "you can't put a price on equality. When it comes to equal rights, cost should not be a factor". He added, "if civil partnerships were made available to heterosexual couples in the UK there would probably be a similar significant take up".We're not so sure. For one thing, there's a difference between short-term and long-term demand - as demonstrated by the number of civil partnerships in the UK which fell quite steadily from 16,106 in 2006 to just 6,795 in 2011.The Datablog covered all the numbers the last time that the Office for National Statistics released data on civil partnerships in July 2012. Here are some of the main statistics that came out of our analysis then (you can find these, and more, in the data link below)The number of people entering a civil partnerships (106,834) was five times the number initially predictedThe average age for entering a civil partnership was just over 40 for men and 38 for women1,768 civil partnerships were dissolved between 2007 and 2011Given that heterosexual couples have the option of marriage (which currently offers a preferable pension to widowed partners than a civil partnership) it seems unlikely that heterosexual civil partnerships will see the same intial uptake. When the Government Equalities Office looked instead to what had happened in other countries such as Argentina, Canada or Iceland where same-sex marriage has been introduced they found that there was "insufficient evidence" on demand. With one exception: the Netherlands.The Netherlands is also a critical case study for the proposed amendment since it has followed a path similar to the one projected for the UK - same sex couples were allowed to enter into registered partnerships from 1998 and in 2001 same sex couples could enter into civil marriages. Since 2001, registered partnerships and civil marriages have been open to same sex and opposite sex couples and they are able to convert between the two.As data from the Netherlands shows, the number of partnership registrations between men and women did increase. But the numbers weren't huge - from 2,847 in 2001 to 7,774 in 2002 (compared to 83,970 marriages between men and women in the same year). In 2009, Tom Freeman and Katherine Doyle began a legal challenge after applying for a civil partnership and being refused. The pair had the backing of Equal Love which, according to its website, also receives the support of MPs and MEPs from the Labour, Conservative, Green and Liberal Democrat parties.You can add your voice to this debate either via Twitter to @MonaChalabi or @GuardianData or by starting a discussion below. Download the data• DATA: download the full

After seeing a Reddit post on the convergence of Miss Korea faces, supposedly due to high rates of plastic surgery, graduate student Jia-Bin Huang analyzed the faces of 20 contestants. Below is a short video of each face slowly transitioning to the other. From the video and pictures it's pretty clear that the photos look similar, but Huang took it a step further with a handful of computer vision techniques to quantify the likeness between faces. And again, the analysis shows similarity between the photos, so the gut reaction is that the contestants are nearly identical. However, you have to assume that the pictures are accurate representations of the contestants, which doesn't seem to pan out at all. It's amazing what some makeup, hair, and photoshop can do. You gotta consider your data source before you make assumptions about what that data represents.

Velocity, not volume is increasingly what determines the hardware and software needs of data-processing organisations• More from the Guardian's series on big dataCritics of big data are picking holes in its validity as a concept, but there is a problem with their arguments around data volume - it is speed, not size, that defines big data in 2013.Big data is among the computing neologisms du jour, and a technology conference in 2013 is rarely considered complete without a smattering of uses, typically accompanied by further volume-related qualifiers (tsunamis of big data being by far the worst offender I have encountered - other suggestions welcome).Despite having been in use for a few years now, settling on just what is meant by big data appears to be a complex task. As is their wont, various computing gurus (that one isn't confined to tech circles) have come up with initialisms to summarise what they see as its key constituent parts.The 'four Vs' definition is probably the most widely recognised - the letters standing for volume, velocity, variety and variability - and from a technical perspective, this is actually a reasonable effort, but a number of otherwise excellent articles currently making the rounds deal only with the first V.A healthy dose of scepticism is a must when dealing with emergent terms in the technology sector, but in this particular case, commentators would do well to delve a little deeper before setting out to dismantle big data as a concept.Last month we re-posted an interesting and well-constructed argument that 'small data' - or data of the volumes most regular analysts, researchers and statisticians are used to dealing with - is actually both more relevant and more useful to the vast majority of organisations than its big cousin.More recently, I read a well-researched article on just how infrequently the world's data powerhouses - citing Facebook and Yahoo! - actually carry out an individual piece of analysis on data that would not fit onto a laptop or desktop machine you could pick up from your local electronics retailer.The points made in both of these articles are eloquently put and there no obvious holes to be found in the arguments' logic. The problem is, both authors dismiss big data on the grounds of volume alone, ignoring the fact that it is speed, not size that is increasingly driving desire for software and hardware improvements at data-processing organisations.The need for genuine real-time results is integral to ever more analytics use cases. There are, of course, industries where gathering, analysing and reacting to data is nothing new - take high frequency trading, for example. But the list is growing, with information security, marketing and telecommunications just three examples of sectors where speed, more than volume, has been identified as a limiting factor.Last week Paul Maritz, CEO of EMC Pivotal, described a marketing paradise in which customers in a store are tracked and served offers while they shop - insights put into action instantly, rather than retroactively, as we currently see with deals offered at the point of sale.Leaving aside for now the numerous privacy and intrusion questions that arise from such a scenario, the message here is that data-centric companies seeking to gain a competitive edge have marked out velocity as the new battleground.In fact it is just as fallacious to consider speed in isolation from volume as it is to do the reverse - the pair form two sides of a speed-data-time triangle. As the values for data and time tend towards infinity and zero respectively, the software and hardware requirements ramp up.In short, once you really consider the technical challenges facing CTOs, data scientists and others embedded in this field, the idea that big data be dismissed as a term because it's not all that big is - however well presented - verging on straw man territory.Which side of the big data debate do you sit on? Join the discussion in the comments below, or

A review of England's fire and rescue services has found it could be saving nearly £200 million per year. Firefighters are already claiming that this is nothing more than a cover for cuts. We take a closer look Get the dataMore data journalism and data visualisations from the GuardianIt must be a Friday: a government report was released that seems to spell out good news for everyone. A document titled 'Facing the Future' set out a dramatic drop over the last decade in the number of fires, fatalities, and related incidents across England.Good news for everyone – except maybe firefighters, as the report's being used to justify substantial proposed cuts and consolidation in the fire and rescue services. The independent review, commissioned by Brandon Lewis, the fire minister, finds that there has been "a massive reduction in emergency incidents in the last decade" but also addresses the potential accusation of cuts head-on (if in somewhat obtuse language) by adding:"I am cognisant of the time in which this review is published, a time of austerity which is likely to continue with downward pressure on public expenditure".Saying that savings of £196m a year are possible for England's fire and rescue services is no small claim – so we've decided to take a look at how that figure is arrived. Is there really that much less to do?Lower risks?According to the report, the fire and rescue services are attending 40% fewer incidents. They published the chart below to demonstrate the trend.When we looked at the annual statistical releases we found the data on fire fatalities that was used to make this graph (page 8). The 15% drop in all fire fatalities does slightly obscure the fact that fatalities from accidental dwelling fires – the 'chip pan' or 'dropped cigarette' fires of family nightmares – fell somewhat less, by only 9%. Next, we looked at the 'incidents' data in this graph - which, according to the key findings in the report have declined by 40%. Well, it appears that 'incidents' refer only to 'fires'. But is there a catch?But firefighters don't only go out to real fires: when a call comes in, they've no way to know if it's a false alarm or not, and of course firefighters engage in other non-fire services, from aiding in road traffic accidents to (apocryphally) rescuing cats in trees. Expanding the range of incidents to cover fire, false alarms, road-traffic accidents and non-road traffic accidents then the total rises from 473,412 incidents in England in 2003 to 908,927. Doing the same for 2012 isn't exactly possible as we only have six months of figures, which total 272,000 for the period of April to September 2012. So, have incidents still fallen? The answer's a slightly unsatisfactory "probably": there could well be more incidents in the winter months than in the summer. But using a simple estimate of 544,000 for the year leaves an overall drop of around 40% in total incidents, not just fires. So it's quite possible the government has a point. New risks?But, then again, they might not. Though it's clear that England's fire and rescue services aren't putting out as many fires, that may not necessarily be reason enough to suggest cutting back. Between April and September 2012, fire and rescue services attended 69,400 non-fire incidents which often require different skills and equipment. Some of these, such as rescue of persons, animal assistance incidents and lift releases have declined since - but several of these incident types are up. Take, for example, the 9,200 incidents of flooding which required the assistance of fire and rescue in England over the space of six months - up 75% from the same period the year before. With meteorologists suggesting that our more turbulent weather is getting harder to predict, it might not be wise to believe that past risk levels accurately indicate future risk levels. It's also entirely possible that the preventative work carried out by fire services is a contributor to falling numbers of fires

With news of $billion-dollar tax evasions joining $billion-dollar bailouts and $trillion dollar deficits in the mind-boggled group mind, we thought it might be a good time to update the Billion Dollar-o-Gram. This time, rather than pain-stakingly hand-drawing our treemap visualisation, we developed some generative, interactive code to render the data. The result, we think, is both beautiful and sickening. VizSweet This ‘TreeMappa’ code is part of our forthcoming VizSweet software – a set of high-end dataviz tools for journalists, presenters and analysts. Find out more. You’ll be seeing more VizSweet over the next few months. It helps us quickly render beautiful datavizzes from any kind of data, so we’ll be using it a lot. (Although part of me snarls when I see our algorithm taking 3 milliseconds to draw what used to take me 3 days to hand-draw!) For nostalgia’s sake, we’ve rendered the old ‘classic’ Billion Dollar-o-Gram in VizSweet-o-vision. Oh and we’ve hidden a little easter-egg in the data-viz. See if you can find it!