Data Visualization

What do emptier waters look like? This web aquarium shows declining fish populations over the past 100 years - and it uses more than 200 datasets to do it. We recommend that you closely analyse the data showing that the population of big fish has been decimated while small fish are now overly abundant. Or else you can leave it on your screen as a digital pet and a grim modern Tamagotchi.Mona Chalabi

In collaboration between USGS, NASA and TIME, Google released a quarter century of satellite imagery to see how the world has changed over time. The images were collected as part of an ongoing joint mission between the USGS and NASA called Landsat. Their satellites have been observing earth from space since the 1970s—with all of the images sent back to Earth and archived on USGS tape drives that look something like this example (courtesy of the USGS). We started working with the USGS in 2009 to make this historic archive of earth imagery available online. Using Google Earth Engine technology, we sifted through 2,068,467 images—a total of 909 terabytes of data—to find the highest-quality pixels (e.g., those without clouds), for every year since 1984 and for every spot on Earth. We then compiled these into enormous planetary images, 1.78 terapixels each, one for each year. Be sure to check out the Timelapse feature on Time.

Heterosexual civil partnerships are on the cards for the UK, thanks to a tabled amendment to government plans on gay marriage. Is there any evidence as to how many people want them?Read the reality check from Shiv Malik hereGet the dataMore data journalism from the GuardianWhen your stance on a subject risks you being branded as homophobic, disrespectful of religion, out of touch or anti-establishment it might be better to argue your case on the basis of cost. And with a subject as sensitive as gay marriage on the table, that's exactly the strategy many MPs appear to be adopting. Or at least indirectly. In fact, the current debate between MPs is about the potential price tag if heterosexual couples were also to be granted civil partnership rights but many suggest that, if passed, the amendment could derail the gay marriage bill.The estimated costs for amending the bill have varied between £90m to £4bn - leading critics such as the Conservative MP Tim Loughton to brand the numbers as "back of fag packet scaremongering". While Shiv Malik is taking a look at how the numbers stack up on the reality check blog, we're considering the number of people the legislation is likely to affect.Peter Tatchell, sponsor of Equal Love, a group campaigning in favour of both same-sex civil marriage and opposite-sex civil partnerships, said "you can't put a price on equality. When it comes to equal rights, cost should not be a factor". He added, "if civil partnerships were made available to heterosexual couples in the UK there would probably be a similar significant take up".We're not so sure. For one thing, there's a difference between short-term and long-term demand - as demonstrated by the number of civil partnerships in the UK which fell quite steadily from 16,106 in 2006 to just 6,795 in 2011.The Datablog covered all the numbers the last time that the Office for National Statistics released data on civil partnerships in July 2012. Here are some of the main statistics that came out of our analysis then (you can find these, and more, in the data link below)The number of people entering a civil partnerships (106,834) was five times the number initially predictedThe average age for entering a civil partnership was just over 40 for men and 38 for women1,768 civil partnerships were dissolved between 2007 and 2011Given that heterosexual couples have the option of marriage (which currently offers a preferable pension to widowed partners than a civil partnership) it seems unlikely that heterosexual civil partnerships will see the same intial uptake. When the Government Equalities Office looked instead to what had happened in other countries such as Argentina, Canada or Iceland where same-sex marriage has been introduced they found that there was "insufficient evidence" on demand. With one exception: the Netherlands.The Netherlands is also a critical case study for the proposed amendment since it has followed a path similar to the one projected for the UK - same sex couples were allowed to enter into registered partnerships from 1998 and in 2001 same sex couples could enter into civil marriages. Since 2001, registered partnerships and civil marriages have been open to same sex and opposite sex couples and they are able to convert between the two.As data from the Netherlands shows, the number of partnership registrations between men and women did increase. But the numbers weren't huge - from 2,847 in 2001 to 7,774 in 2002 (compared to 83,970 marriages between men and women in the same year). In 2009, Tom Freeman and Katherine Doyle began a legal challenge after applying for a civil partnership and being refused. The pair had the backing of Equal Love which, according to its website, also receives the support of MPs and MEPs from the Labour, Conservative, Green and Liberal Democrat parties.You can add your voice to this debate either via Twitter to @MonaChalabi or @GuardianData or by starting a discussion below. Download the data• DATA: download the full

After seeing a Reddit post on the convergence of Miss Korea faces, supposedly due to high rates of plastic surgery, graduate student Jia-Bin Huang analyzed the faces of 20 contestants. Below is a short video of each face slowly transitioning to the other. From the video and pictures it's pretty clear that the photos look similar, but Huang took it a step further with a handful of computer vision techniques to quantify the likeness between faces. And again, the analysis shows similarity between the photos, so the gut reaction is that the contestants are nearly identical. However, you have to assume that the pictures are accurate representations of the contestants, which doesn't seem to pan out at all. It's amazing what some makeup, hair, and photoshop can do. You gotta consider your data source before you make assumptions about what that data represents.

Velocity, not volume is increasingly what determines the hardware and software needs of data-processing organisations• More from the Guardian's series on big dataCritics of big data are picking holes in its validity as a concept, but there is a problem with their arguments around data volume - it is speed, not size, that defines big data in 2013.Big data is among the computing neologisms du jour, and a technology conference in 2013 is rarely considered complete without a smattering of uses, typically accompanied by further volume-related qualifiers (tsunamis of big data being by far the worst offender I have encountered - other suggestions welcome).Despite having been in use for a few years now, settling on just what is meant by big data appears to be a complex task. As is their wont, various computing gurus (that one isn't confined to tech circles) have come up with initialisms to summarise what they see as its key constituent parts.The 'four Vs' definition is probably the most widely recognised - the letters standing for volume, velocity, variety and variability - and from a technical perspective, this is actually a reasonable effort, but a number of otherwise excellent articles currently making the rounds deal only with the first V.A healthy dose of scepticism is a must when dealing with emergent terms in the technology sector, but in this particular case, commentators would do well to delve a little deeper before setting out to dismantle big data as a concept.Last month we re-posted an interesting and well-constructed argument that 'small data' - or data of the volumes most regular analysts, researchers and statisticians are used to dealing with - is actually both more relevant and more useful to the vast majority of organisations than its big cousin.More recently, I read a well-researched article on just how infrequently the world's data powerhouses - citing Facebook and Yahoo! - actually carry out an individual piece of analysis on data that would not fit onto a laptop or desktop machine you could pick up from your local electronics retailer.The points made in both of these articles are eloquently put and there no obvious holes to be found in the arguments' logic. The problem is, both authors dismiss big data on the grounds of volume alone, ignoring the fact that it is speed, not size that is increasingly driving desire for software and hardware improvements at data-processing organisations.The need for genuine real-time results is integral to ever more analytics use cases. There are, of course, industries where gathering, analysing and reacting to data is nothing new - take high frequency trading, for example. But the list is growing, with information security, marketing and telecommunications just three examples of sectors where speed, more than volume, has been identified as a limiting factor.Last week Paul Maritz, CEO of EMC Pivotal, described a marketing paradise in which customers in a store are tracked and served offers while they shop - insights put into action instantly, rather than retroactively, as we currently see with deals offered at the point of sale.Leaving aside for now the numerous privacy and intrusion questions that arise from such a scenario, the message here is that data-centric companies seeking to gain a competitive edge have marked out velocity as the new battleground.In fact it is just as fallacious to consider speed in isolation from volume as it is to do the reverse - the pair form two sides of a speed-data-time triangle. As the values for data and time tend towards infinity and zero respectively, the software and hardware requirements ramp up.In short, once you really consider the technical challenges facing CTOs, data scientists and others embedded in this field, the idea that big data be dismissed as a term because it's not all that big is - however well presented - verging on straw man territory.Which side of the big data debate do you sit on? Join the discussion in the comments below, or

A review of England's fire and rescue services has found it could be saving nearly £200 million per year. Firefighters are already claiming that this is nothing more than a cover for cuts. We take a closer look Get the dataMore data journalism and data visualisations from the GuardianIt must be a Friday: a government report was released that seems to spell out good news for everyone. A document titled 'Facing the Future' set out a dramatic drop over the last decade in the number of fires, fatalities, and related incidents across England.Good news for everyone – except maybe firefighters, as the report's being used to justify substantial proposed cuts and consolidation in the fire and rescue services. The independent review, commissioned by Brandon Lewis, the fire minister, finds that there has been "a massive reduction in emergency incidents in the last decade" but also addresses the potential accusation of cuts head-on (if in somewhat obtuse language) by adding:"I am cognisant of the time in which this review is published, a time of austerity which is likely to continue with downward pressure on public expenditure".Saying that savings of £196m a year are possible for England's fire and rescue services is no small claim – so we've decided to take a look at how that figure is arrived. Is there really that much less to do?Lower risks?According to the report, the fire and rescue services are attending 40% fewer incidents. They published the chart below to demonstrate the trend.When we looked at the annual statistical releases we found the data on fire fatalities that was used to make this graph (page 8). The 15% drop in all fire fatalities does slightly obscure the fact that fatalities from accidental dwelling fires – the 'chip pan' or 'dropped cigarette' fires of family nightmares – fell somewhat less, by only 9%. Next, we looked at the 'incidents' data in this graph - which, according to the key findings in the report have declined by 40%. Well, it appears that 'incidents' refer only to 'fires'. But is there a catch?But firefighters don't only go out to real fires: when a call comes in, they've no way to know if it's a false alarm or not, and of course firefighters engage in other non-fire services, from aiding in road traffic accidents to (apocryphally) rescuing cats in trees. Expanding the range of incidents to cover fire, false alarms, road-traffic accidents and non-road traffic accidents then the total rises from 473,412 incidents in England in 2003 to 908,927. Doing the same for 2012 isn't exactly possible as we only have six months of figures, which total 272,000 for the period of April to September 2012. So, have incidents still fallen? The answer's a slightly unsatisfactory "probably": there could well be more incidents in the winter months than in the summer. But using a simple estimate of 544,000 for the year leaves an overall drop of around 40% in total incidents, not just fires. So it's quite possible the government has a point. New risks?But, then again, they might not. Though it's clear that England's fire and rescue services aren't putting out as many fires, that may not necessarily be reason enough to suggest cutting back. Between April and September 2012, fire and rescue services attended 69,400 non-fire incidents which often require different skills and equipment. Some of these, such as rescue of persons, animal assistance incidents and lift releases have declined since - but several of these incident types are up. Take, for example, the 9,200 incidents of flooding which required the assistance of fire and rescue in England over the space of six months - up 75% from the same period the year before. With meteorologists suggesting that our more turbulent weather is getting harder to predict, it might not be wise to believe that past risk levels accurately indicate future risk levels. It's also entirely possible that the preventative work carried out by fire services is a contributor to falling numbers of fires

With news of $billion-dollar tax evasions joining $billion-dollar bailouts and $trillion dollar deficits in the mind-boggled group mind, we thought it might be a good time to update the Billion Dollar-o-Gram. This time, rather than pain-stakingly hand-drawing our treemap visualisation, we developed some generative, interactive code to render the data. The result, we think, is both beautiful and sickening. VizSweet This ‘TreeMappa’ code is part of our forthcoming VizSweet software – a set of high-end dataviz tools for journalists, presenters and analysts. Find out more. You’ll be seeing more VizSweet over the next few months. It helps us quickly render beautiful datavizzes from any kind of data, so we’ll be using it a lot. (Although part of me snarls when I see our algorithm taking 3 milliseconds to draw what used to take me 3 days to hand-draw!) For nostalgia’s sake, we’ve rendered the old ‘classic’ Billion Dollar-o-Gram in VizSweet-o-vision. Oh and we’ve hidden a little easter-egg in the data-viz. See if you can find it!

The Eurovision 2013 final is upon us, but who does the data say will win – and will Bonnie Tyler be totally eclipsed?• Get the voting dataIt's finally here: the eve of the kitschest night of the year: the Eurovision song contest. While for most the fun comes tomorrow – who'll have the strangest accent? Will the hosts be dire? Will anyone actually give the UK any points? – for us at the Datablog the real fun, of course, is in the stats.So, all week we've done nothing more than work out how to predict who's going to win this year's contest. And we're going to – boldly – set out what we reckon below. But any gambling losses as a result of our predictions are, we're afraid, entirely on your own heads.We've tried to predict the Eurovision before, in 2011, and didn't do too badly: the eventual winner was in our predicted top three. This year, though, we've got two rival systems – one from our data editor, James, and the other from Margot Huysman.Prediction one: the cynic's view – JamesBy now, even infants aren't naïve enough to think the Eurovision Song Contest has anything to do with music. It's all about politics: who likes who, who's in, who's out – and who might cut off the oil supply this winter if they don't get douze points.Crunching scoring data from the last ten years corroborates the cynics' view (as most things do): voting blocs seem to be pretty real, and points given correlate with points received. We've put all of this voting information in the interactive graphic below. The eastern European voting bloc in particular seems quite strong:So, what does that mean for the winner? At the extreme, it suggests we could ignore the song entirely, and just look at the voting patterns for the last decade: the countries who score well one year should score pretty well the next (with the song being mere random noise).That's what we've shown in the table below – which averages scores over the last decade, then removes countries which haven't reached the final for the last five years.It seems to bode well in particular for Azerbaijan and Russia on Saturday night, but then as the next two top tips haven't even made the final (gulp), the third tip has to be Greece.But are they really all that popular in the Eurozone right now? And could they afford the hosting bill if they win?Prediction two: the song matters – MargotUntil 2004, qualification for the Eurovision contest was based on the average points received over the five previous years. However, that year, the system was dropped in favour of semi-finals.The songs performed in the semi-finals are exactly the same as the ones performed during the final, and apart from a longer running order, all three nights are quite similar. If votes are based on preferences, then the songs that are popular in the semi-finals are popular in the final show as well.Ever since the institution of the semi-final system, the winner of the competition has been present in the list of top songs in the semis. The only time this has not happened was when Germany won in 2010.The data looks good for using the semis to predict the result: for the nerds, about 55% of the score in the finals is predicted by the score in the semis. In other words, you're really quite likely to score well in the finals if you do in the semis. So this is a brilliant way of predicting the final!A slight flaw in this method is that it does not account for countries that are directly qualified for the final, which are France, the UK, Germany, Spain and Italy. However, none of these countries – except, as we said earlier, Germany – have won in the last nine years.There's maybe a bigger problem, though. Eurovision themselves have also realised the semi-finals help prediction a lot. And so in order to "maintain the suspense going until the last minute", they've hidden the results. But if you had access to the points, looking at the top scorers in the semi-finals could give you a good indication of who will be in the top three. This being the

The Datablog is trying to predict the winner of Eurovision 2013 – but we need one more bit of data: who did you think did best in the semi-finals?The results – and the rest of our predictive data – will be posted on the Datablog on Friday afternoon Eurovision 2013EurovisionJames Ballguardian.co.uk © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds

Deadspin made a straightforward map that shows the highest paid public employee in each state. Based on data drawn from media reports and state salary databases, the ranks of the highest-paid active public employees include 27 football coaches, 13 basketball coaches, one hockey coach, and 10 dorks who aren't even in charge of a team.