.NET (299) administrative (42) Ajax (42) AngularJS (2) ASP.NET (146) bicycle (2) books (199) browser (10) C# (135) cars (1) chess (29) CodePlex (10) Coma (8) database (55) deployment (3) Entity Framework (2) essay (119) flash/shockwave (2) flex (1) food (3) friend (2) game (21) idea (5) IIS (8) javascript (85) LInQ (2) Linux (6) management (4) manga (45) misc (712) mobile (1) movies (101) MsAccess (1) murder (2) music (64) mysql (1) news (101) NuGet (1) permanent (1) personal (67) PHP (1) physics (2) picture (335) places (12) politics (15) programming (520) question (2) rant (123) religion (3) science (44) Sharepoint (3) software (59) space (1) T4 (2) technology (13) Test Driven Development (4) translation (2) VB (2) video (104) Visual Studio (45) web design (47) Windows API (8) Windows Forms (3) Windows Server (6) WPF/Silverlight (63) XML (12)

Saturday, May 23, 2015

TV Shows so shitty, that even I couldn't stand watching them

As some of you know, I am watching a lot of TV series, including cop and lawyer stuff, childish fantasy and sci-fi series, pirate and musketeer crap, etc. So when I say that I couldn't watch something, you should heed me. There are also some shows that I actually can stand watching, but barely, usually using the fast forward to watch an episode in ten minutes or less.
Here is a list of just such shows:
  • Lost - I think it all started with Lost. After a brilliant pilot episode, it got into 1001 Arabian Nights territory and became ridiculously unwatchable. While it wasn't a truly bad show (some people actually enjoyed all 6 seasons) I couldn't watch it more than 2 seasons. Calm down, the rest of the shows will be more recent.
  • The Messengers - every once in a while the American Bible thumpers convince producers to create religion based TV series or movies. While every possible superstition has and will be used in films, the Christian ones are standalone - in their own cinematic universe, if you will - and take themselves seriously every time. Messengers is one of these, and I couldn't watch more than 2 episodes before I gave up. It's (again) about the upcoming apocalypse and end of days and these 8 idiots are given superpowers to convince God that we are still worth it, while Satan goes around doing silly things. It was not only about the most tabloid part of (a part of) the bible, but it was acted and scripted badly. Ugh!
  • Dig - This relates more to Hebrew religion and feels like a TV series take on the Da Vinci Code, with clones of Jesus (I think), FBI policemen running around in Israel doing whatever they feel like, shady cabals and shady opposing cabals. It could have been interesting and I admit that I was drawn into it by the acting and production values. Unfortunately the script was a convoluted mess that had no way of improving. They never learn to ease into it, present a little bit of the story, add another element later on, make people feel rewarded for continuing to watch the series. Anyway, four or five episodes in, I gave up.
  • iZombie - Oh, you have to love this: young blonde accidentally gets turned into a zombie and gets a job at the morgue in order to have access to a fresh supply of brains. While consuming said brains, she gets visions of what the brain's owner had lived and thus helps a police detective to find out who murdered them. It combines police procedurals with zombies, comedy, romance and drama. Talking of messes, this is as messy as they come. I watched a few episodes before I slapped myself "what the hell are you doing watching this swill?"
  • Eye Candy - A darker show about a girl using her hacker tech skills to try and catch the serial abductor and killer of young women, including her sister. Yet the killer is just as able and a game of cat and mouse ensues. It started well, then went into pop culture territory. Besides, she continues on only because the killer refuses to murder her and get on with his job. Something about the way the film was made or the main actress or the ridiculous plot made it impossible for me to get past the second episode.
  • Allegiance - It is strange to put this series here, as I watched all the episodes so far. However, it was hope that kept me going, not the quality of the show. While it is about Russian spies in the US, living as a family, while their unknowing son is recruited into the FBI, while the antagonists are interesting, the story goes so much over the top that it becomes overcheesy. That means it starts as cheesy, then it starts to smell. Just do yourself a favor and do not watch it.
  • Suits - Yet another flashy show that I kept watching for no good reason. It is about lawyers. Young, cocky, caught in personal intrigues and competing with one archnemesis after another, while occasionally engaging in some silly romance. It makes no sense, although I have to admit that I liked the feel of the show. It just goes nowhere. They get the mood right, the actors, the characters, then they make a joke of the storyline and script.
  • Helix - Helix is the new Lost! It starts with a subject close to my heart: killer viruses, and then it crashes into la-la-land. Viruses that make people immortal and strong so they form a world controlling organization, CDC teams that act like they are terrorist splinter cells with no support, no rules, no communication, strains of the virus that turns people into The Strain like vampires, religious cults that are also genetic scientists living on their own island, etc. It starts as something, goes into something else, switches back. It's like someone was undecided what to write about and they wrote about everything then they just drew lines between the unrelated subjects and wrote a script. It is a complete and utter waster of time.
  • Elementary - I watched two seasons of this until I admitted to myself that there is no point in continuing. I like both Johnny Lee Miller and Lucy Liu, I love Arthur Conan Doyle's Sherlock Holmes character and so a modern Sherlock should have been something I would at least enjoy. Unfortunately, it shortly turns into another "gifted person helps the New York police" show and one can feel when the writers just completely stopped giving a damn, if they ever did before beyond pushing the pilot.
  • Scorpion - Talking about gifted persons helping the police, this had a promising concept: a genius young man creates a sort of a thinktank of gifted people then starts helping his FBI parent surrogate. While the characters were brilliant, the people writing their script obviously were not. It took a few episodes to completely fail and another ten for me to just give up hope.
  • The Librarians - God, what a complete and utter mess. A TV show spinoff of the ridiculous Librarian movies, it was completely stupid. I mean, the movies were masterpieces compared to the writing for this show. Stay away!
  • The Newsroom - This is a show that has ended and I watched all of its episodes. However, it was under duress. My wife liked it, but then I could see that she couldn't watch it anymore. I watched the last mini-season just in order to tell her it was not worth watching. It's about a news organisation that fights for its moral integrity. It would have been fun if not for the hyperactive writers that thought people could normally carry conversations like machineguns shoot bullets and the ridiculous romantic interludes that made no sense for anyone but a New Yorker raised on Woody Allen movies.
  • State of Affairs - I already ranted enough about this piece of propagandistic crap. The plot was stupid, the script was stupid, the actors couldn't give a damn. It's just bad!
  • The 100 - In theory I am still watching this young adult sci-fi TV show. In reality, I can't make myself start watching the second season. Why? Because the sci, as usual, gets completely negated by the fi. Add to this hormone induced behaviour of teenage kids and savage barbarians and you get a load of messy crap. It started really well, though. Too bad.
  • Extant - Another sci-fi show in which they dumped randomly whatever was hot at the moment: aliens, artificial intelligence, the Japanese guy from Helix, etc. Add to this the excruciating acting skills of Halle Berry and you get something unwatchable.
  • Stalker - This is one of the more interesting police procedurals. It involves a team that is tasked to handle stalkers. The new hire of the agency may be a stalker himself. The lead actors are people that I respect. However the story was so banal and lacking anything that showed any concern for the lives of the characters or victims, it was so formulaic that it ended up being something I regret having watched.
  • Arrow - OK, I am still watching The Arrow, but I do it on fast forward. It started as a fun comic book TV show with a vigilante dressed in green killing assholes with the use of arrows. It then started growing and expanding until it reached a full team of heroes, with impossible situations, history, skill learning curves, ridiculous villains. It culminated with John Barrowman having a line in the show in which he tells three other characters he is the better man for a job because he is the better at acting. Which is true, but it just shows that even the scriptwriters are tired of the mediocre talents in this mediocre show. I keep watching it because it is set in the same universe as The Flash, which is the better TV show, even if not by much.
  • Gracepoint - When Americans are redoing a show with the same script as the original show from a different country I snicker. Stoopeed Americaens, they can't read subtitles. But Gracepoint upped the ante: they redid Broadchurch, a British TV show. Not only that, they cast the same actor as the main character. I love David Tennant, but I can't possibly condone watching this. Anyway, it was cancelled after a season, and then Broadchurh continued for a second season, also starring Tennant.
  • Sleepy Hollow - An entire TV series based on the Legend of the Sleepy Hollow. Starring a colonial Mason that fought for and knew the forefathers of the United States, resurrected in our times and fighting along a black female police officer to stop the End of Times. What can possibly go wrong here? Yet it wasn't that clear cut. The comedy was fun, the original drama had impact, the romance was believable and John Noble, like in Fringe, was awesome. However the storyline, like in Fringe, descended into a chaotic mess that makes no sense and no one could ever care for. Too bad, since it wasn't a complete waste of time, especially at the beginning.
  • Madam Secretary - Just as with State of Affairs, I've ranted enough about it. It's a pure propaganda piece, with no artistic value whatsoever.
  • Haven - It started as a nice supernatural police procedural with a sexy FBI agent coming in a town where people get dangerous "troubles" which act like superpowers out of control. I kept waiting for it to get anywhere, instead it just went more and more overdramatic and adding storylines that made no sense and in the end turned into a supernatural 90120. Too bad, since it started well and has some actors that I really like.
  • Forever - Forever boring, this show is about a British doctor who never ages and, whenever gets killed, he finds himself alive and well in a nearby body of water, completely naked. And what does he do with that? He helps the police, as a forensic medic. Just as a general rule: avoid the gifted people helping the police catch bad guys shows. They never end well and they are as well propaganda pieces.
  • The Lottery - Badly made show. Bad casting, bad acting, bad scripting. The plot: people stop having children, embryos stop getting fertilized and a woman doctor finds a way. Obviously, the powers that be want to control this in a market of dwindling supplies. Could have been nice, probably, although shows about children and parently feelings never do well in the long run, but it was pure shit from the beginning.
  • Taxy Brooklyn - Someone tried to combine the French Taxi storyline with a police procedural. Sexy ass police detective, a Black French taxi driver, some intrigue and of course the obligatory connection with an old case involving a relative. But it was bad. The actors did what they could with lazy writing, but they couldn't save it.
  • The Witches of East End - Some of the most beautiful women in acting get together with ridiculously good looking men in order to act a really stupid Dynasty-like plot. Just like similarly afflicted The Originals, it is a complete waste of time, unless you are a hormonal teenager and you need something to jack off to.
  • The Leftovers - Trying to express the horrible anguish of being among the ones left on Earth after 2% of the population randomly disappears. Of course, it hints of another Bible concept: the Rapture. It felt to me overly dramatic, with people just doing stuff for no good reason and overly depressing as well for equally no good reason. It seemed to be well acted, but I just couldn't watch it.
  • Resurrection - an American remake after Les Revenants, it missed the point completely. Ended up being this drawn out mystery that led nowhere, with really unsympathetic characters and without any of the real tension from the French series.
  • Da Vinci's Demons - I know, I am still watching it, but not because it is worth watching. It is about a young Da Vinci who does everything from saving Florence to saving Italy and then going to the Americas (way before it was discovered) to look for clues about his mother, who was there previously. It is a fun to watch show, but it really goes nowhere. I liked the acting and the production values, though.
  • Crisis - An intriguing concept: some very resourceful people kidnap a lot of high profile adolescent kids, including the son of the president of the United States, then they use the parents to achieve their goals. While the show was well done, the story was just a lazy fantasy trying to surf the edge of suspension of disbelief. Again, it goes nowhere.
  • The Tomorrow People - An American take of an old British children's TV show that wasn't good to begin with, it only served to promote certain actors to better TV shows. It used the common tropes: child with mysterious father who dies doing something awesome, mother unknowing, uncle being involved somehow, evil organizations, rebellious teenagers fighting them, etc. It just collapses under the ridiculous ideas that they introduce to appease a boring viewership.
  • The Originals - Some spin-off from a vampire show that I never watched, it is about pompous asses acting even more pompous because they are the original vampires. Werewolves and witches are thrown in the mix, but the only worth mentioning attribute of the show is the pompousness. Which is annoying. So don't watch it.
  • Killer Women - Trying to get on the diversity bandwagon and get some easy money, it was supposed to be a police procedural with tough women. Didn't pan out. Was cancelled.
  • Bitten - One of the most beautiful actresses around is a werewolf. Instead of being about her biting more than she can chew (heh heh), it is about another clan/royalty obnoxiousness, while the world is blissfully unaware and rivals are trying to take power or kill them or whatever. Do you know why people don't officially acknowledge vampires and werewolves? Because they act all annoying and overbearing while being completely idiotic. Nobody wants to know these people!

OK, as an interesting experiment on my preferences vs the public preferences, I will tell you how many of the shows above were cancelled and how many were not. Ready?

Completed (meaning not cancelled, but finishing their story): 2
Cancelled: 16
Still Running or undecided: 15

So there you go, studios think 50% of the shows that I can't stand should be running and they are basing their decisions on public choice. What do you guys think?

Monday, May 18, 2015

Encapsulating indexes or creating a database that allows for fast inserts and updates as well as indexed queries

Relational databases have the great advantage of being able to use indexes. These are constructs that trade space and the speed of inserts and updates for query speed. This introduces a problem: what do you do when you have a high input and high output application? How can you make your database fast for both changes and queries?

The traditional solution is a write-database and a read-database. There are background services that ensure that the read-database is updated as fast as necessary with the new data, most of the time also doing extra computation and caching for the information one needs extracted. OLAP systems are a common solution, where the aggregated data required by various reports is computed and stored, ready to be read by your applications. I am not saying this is a bad idea, in fact I am saying this is the best idea for this scenario. The problem that I have with it is that you can hardly automate the process. You need to know what you want to read, you need to write the software to create the data and aggregate it into what you need.

So I decided to try to build a system that obeys the following rules:
  1. The speed of insert and update operations needs to be unhindered by indexes. Indeed, changes to the original data should be avoided.
  2. The select operation need to benefit from the indexing system.
  3. The system must be able to determine by itself the data structures needed to cover the first two rules.
  4. Optionally, there should be a way to encapsulate this new behaviour into a separate data structure from the original database.

In order to insert and update as fast as possible, I needed tables that only had primary keys, identity integers rather than uniqueidentifiers, as they lead to fragmentation of clustered indexes. In order to not only index the columns that are used in where and join conditions, but also encapsulate the behaviour in some other data structure, I decided to create shadow tables with the columns that I needed for querying. These tables I would then index in order to improve selects. The connection between the original insert table and the select table would be made via primary keys and the synchronization between the two types of tables would be done in the background, whenever needed. Best of all, analysis on execution plans could automatically determine the columns needed for this system and create the tables and indexes required, then suggest improvements on the observed queries.

In order to test my theory I created the following tables:
  • OriginalTable - with a primary key identity ingeniously called Id and two other columns called Latitude and Longitude, containing random decimal(18,6) values
  • LocationIndexTable - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and two Latitude and Longitude columns that are indexed by the same non clustered index
  • LocationIndexTable2 - with a refId integer primary key column that links to the OriginalTable Id - without being a foreign key - and a locId integer column that links to another table called LocationTable and has its own non clustered index
  • LocationTable - with a primary key identity column and Latitude and Longitude columns. I know that this looks identical to LocationIndexTable, but this is the more complex case where there are multiple records with the same location. LocationTable holds the distinct Location and Latitude values
  • LocationIndexTable3 - with a refId integer column that links to the OriginalTable Id and is indexed by its own nonclustered index - without being a foreign key - and two Latitude and Longitude columns that are indexed by a clustered index

With a number of 77179072 original table rows, I attempted the queries for each case, careful to clean the cache and memory buffers before that. Here are the results:
  • SELECT count(1) FROM OriginalTable WHERE Latitude BETWEEN 45.5 AND 46 AND Longitude BETWEEN 8.5 AND 9 - no indexes whatsoever. Result: 30 seconds
  • SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 17 seconds
  • SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable2 lit2 ON lit2.RefId=ot.Id INNER JOIN LocationTable lt ON lit2.LocId=lt.Id WHERE lt.Latitude BETWEEN 45.5 AND 46 AND lt.Longitude BETWEEN 8.5 AND 9. Result: 41 seconds
  • SELECT count(1) FROM OriginalTable ot INNER JOIN LocationIndexTable3 lit ON lit.RefId=ot.Id WHERE lit.Latitude BETWEEN 45.5 AND 46 AND lit.Longitude BETWEEN 8.5 AND 9. Result: 22 seconds

Unexpectedly for me, the most comfortable solution also seems the faster. Indeed, one issue is that I didn't have duplicate location data in the database, so there was no advantage in adding a new table to link locations to the original table. That means that in cases where the indexed data has many duplicates, it might be advantageous to experiment with a "distinct" table, although indexing should take this case into account, as well. The clustered index is slower than the unclustered one, that was a surprise. Also, the indexing just made the query twice as fast - I had expected more. Of course, all this benchmarking was done after deleting the cache and buffers with the commands DBCC FREEPROCCACHE; DBCC DROPCLEANBUFFERS. It is interesting to see how fast they queries go without this clearing. The first unindexed query takes 3 or 4 seconds, while all the others take less than 2.

There is one more thing that needs to be addressed: moving these tables into another database. Are the indexes just as fast? They should be, but we must test interdatabase communication. This would allow to move the entire system outside a target database, truly encapsulated, really adding value to a database that remains unaffected. My tests show the following results: 28, 18, 29 and 18 seconds, respectively. Yes, you saw that right, the speed of the joins when the indexed databases are in another database on the same server seems faster! Just to make sure I reran the original tests on the same database and I got approximately the same results: 29,19,43 and 24, respectively. The only thing I didn't try (and I don't intend to) is to create primary-foreign key associations, as this means modifying the original tables, something I wish to avoid.

So, after all this I believe my idea has been validated. A lot more work has to be done in order to automate this system, but at least a no-effort manual solution is possible. Let's recap:
  1. The speed of row inserts and updates remains unchanged: the only index is the original primary key identity integer key that should always exist in a table anyway.
  2. The speed of selects is improved by creating tables that have an integer primary key that links to the original table, and only the columns used in queries, over which indexes are created.
  3. Queries can be used to determine the columns needed to index. Even better, computed columns can be used instead of the original columns, which adds a little more performance.
  4. Encapsulation is achieved by not only creating other tables for reading, but also moving these tables into another database, after which, unexpectedly, the performance is even better.

The end result is a system similar to indexing, but which takes space in another database and which is updated on demand, not when the system deems it necessary. Any SQL statements that worked before will continue to work unaffected, but faster alternatives for selects will become available. Overall, I am very satisfied, although I had expected better performance improvements than observed.

Friday, May 01, 2015

The spy TV shows running right now

There are these time periods in which movies and TV shows start to follow a common theme. I don't know exactly why that happens, but I suspect it's about Hollywood corporate managers stealing everything that seems to work and getting to copy each other. We seem to be living under the sign of the Spy right about now and in this blog post I will be discussing the following TV series (listed in alphabetical order):
  • Allegiance - a show about a CIA analyst who is the unknowing son of a pair of Russian KGB spies. He is also a kind of idiot savant which can remember a lot of things and make connections that no one can. It gets really hard to believe soon as it mashes up family drama with evil SVR (former KGB) machinations, CIA and FBI operations, traitors and double agents, hired assassins, plots to bring down the United States, and so on. It's one of the lesser spy shows and I don't recommend it.
  • The Americans - I believe this is the gem of the lot. The protagonists are two fake Americans, in reality KGB agents. They live like an American couple, with the mandatory two children and their neighbor is an FBI agent in the very taskforce that is supposed to catch people like them. The acting is very good and the show is a serious one, going only occasionally over the top. Also the lovely Keri Russell makes it very appealing as well :)
  • The Assets - The Assets has ended after a single season, with the successful completion of the main plot. It was a strange mix of really good acting and rather weak characterisation. It followed the story of Aldrich Ames, a famous disgruntled CIA operative who sold American secrets to the Russian KGB. As it is a true story, the entire plot was not terribly captivating, but the problem for me was with the characters.
  • Turn - About the spy ring of the American rebels against the British during the Washington era, it is an intricate and interesting story. I like most of the characters, even the villains, which is rare in films these days. I particularly like Burn Gorman in this, even if he plays this British buffoon that understands nothing of what is going on around him.
  • X Company - Set during World Word II, it is about a company of spies working in the German occupied territories. Their newest recruit is a guy who can remember everything he sees, but is a little weird. Sound familiar? Heh

From the list above, The Americans shines bright. Good acting, good directing, but also a dedication to show the political and social issues of the time. From the ones here, it is the only one who I feel can teach one something about history and about human nature in general. Turn is also rather good, but it is too reliant on a number of characters that keep ending up crossing paths in implausible situations, and many times goes a little over the top. The actors are really good in this one, though, and it is worth watching for sure. X Company is more like the typical WWII movie, with the evil Germans and the brave resistance fighters. They do attempt occasionally to enter the grey areas, like good German officers, conflicted collaborators or remorseful secret agents who have to compromise between human values and the mission, but it is not enough. When the beautiful team leader played by Evelyne Brochu reveals that she is half Jewish and the whole story goes into Holocaust territory you can feel the show going south. The Assets was brave in not employing only beautiful actors. The main character was ugly and weak and constantly fearing discovery, hard to empathize with. But so were the people who were trying to catch him, led by an obsessive woman that acted like a blood hound the entire time. Not being able to make an emotional connection to any of the characters hurt the success of this otherwise interesting story. This leaves Allegiance, which feels a lot like a fairy tale combined with a police "special" procedural (you know the type: special person helps police solve crimes) which incidentally features spies. While I cannot really recommend X Company, Allegiance is the only one that I would recommend against.

Worth mentioning is that three of these shows are based on real events. The Assets follows the exploits of Aldrich Ames, X Company is loosely based on Camp X and Turn links to the story of the Culper Ring.

That's it for pure spy shows. I am not putting in this list shows like Homeland or Legends, which also show undercover operatives, but are not specifically aimed at spying as the main subject of the plot. Tell me what you think.

Thursday, April 30, 2015

The Black Company (Black Company book I), by Glen Cook

Book cover Another fantasy audio book, The Black Company is the first of ten of the ongoing series with the same name. Glen Cook probably thought about it for a while before writing it, because you can see that the first three books in the saga were published mere months apart (*cough*R.R.Martin*cough*). I was attracted by the story, because it seemed similar to my favorite Malazan Book of the Fallen series: a mercenary company fighting under the flag of powerful god-like magical beings that want to conquer the world. And in part the plot and characters were satisfactory. Alas, the many details and focus on small things like what games they played to pass time or what each character was like or maybe the narration style... I don't know; it all made it hard to focus on the story. I felt like the book could (and should) have been half in size, and better for it.

The main character is the doctor and analyst of the company, Croaker. Before your filthy mind thinks of other things, analyst in this context means he is a keeper of records, a chronicler of the company's history and, occasionally, the person who draws parallels between past experiences and current events. Croaker is an inquisitive person, often risking his personal safety to unlock a riddle or reveal a secret. Unlike Malazan, he is the lead character, through and through, and the company itself with its many soldiers and camaraderie just the backdrop.

Plot follows the Black Company in employment of The Lady, a powerful magician who wants to conquer the world, fighting the Rebel, a group of lesser magicians who have gathered the people of the land in response. Croaker has romantic fantasies about her, but throughout the book he discovers that his affections are misdirected, even if a strange relationship develops between them. Most of the story is about various battles, painstakingly (and painfully) described, only to be followed by the occasional interesting bit of character and plot development. In fact, I was kind of annoyed when I read the book summary on Wikipedia and I couldn't think of much that was left out. I mean, it's a big book.

The bottom line is that I am unsure if I want to listen to the rest of the series. While I can't say it was a bad book, or that is had weak plot or characters, I am reluctant to go through all that again for nine times. Basically I feel that the characters were never made empathetic enough, at least for me, and in the end all I cared about was what was going to happen next and how it would all end. In that case I would be better off reading synopses rather than listening to the entire thing. The overall structure of the story, though, has a lot of potential and I don't know if the series would not become really cool afterwards. To make the final parallel with Malazan, the first book in that series was not really making people eager to read the next, but it turned out to be great overall. I guess time will tell.

Tuesday, April 28, 2015

Mobile operators transparently change the content you download from the Internet

... and I don't mean something like injecting ads; I mean they modify the images you download and the pages that you read. They do that without telling you, under the umbrella of "improving your browsing experience". Let me give you some examples.

Today I copied two image files on a server: a JPEG and a PNG file. When downloaded via a normal network connection, I was getting the original file, about 50KB in size. When downloaded via 3G the image was different! In the jpeg case the file was smaller by a few hundred bytes and in the case of the PNG the file was actually bigger than the original. What was worse, the metadata information in those photos, like the software used to compress it, for example, was completely lost.

I couldn't believe my eyes. I strongly believe that what you ask for from the Internet you should get. This may not have been obvious for someone downloading the images in order to see them, but I was actually conducting a test that depended on the exact size of the file I was downloading. This practice seems to be widespread, but when Googling for it very few links pop up, showing that people mostly have no idea that it happens. In this operator's case, they seem to only change images, but people on the Internet tell stories of bundling CSS and Javascript files inside HTML files, or removing comments from either of them.

The morality of this is dubious at best, in my view it should be illegal, however things are worse than that: this behavior breaks functionality in existing sites. How can you possibly guarantee that your application works as expected when mobile operators (and I guess any ISP) can change your content arbitrarily and without the possibility to opt out? It's like that joke with the boyscout explaining at camp that little old ladies are hardier than one might think, as they squirm and shout and hit you when you try to cross them the street and the instructor soon finds out that it never occurred to the boy that he should first make sure they want to get to the other side.

Here are some links regarding this, just to make sure people can find them more easily:

Mobile operators altering (and breaking) web content
Should mobile operators be free to modify content they deliver?
Mobile Proxy Cache content modification by O2
O2 UK mobile users - your operator is breaking this site for you.
Prevent mobile website image compression over 3G
Get rid of image compression on O2′s network
ByteMobile Adaptive Traffic Management

From the last link you can see that they are caching and modifying even movies, through practices like giving you a lower rate movie or caching a version with a lower resolution. They do this in the name of delivering you video content compressed with a format and codec that your device can safely open.

A solution for this? Encryption. Using HTTPS prevents access to the content from a third party. HTTPS is becoming more and more used, as the hardware requirements for its implementation become less restrictive and with the many revelations about government scrutiny of Internet communications. However, you might be interested to know that mobile operators are feeling threatened by it. Articles from their point of view decry the "threat" of encryption and the solutions against it! The terrible impact of encryption is seen as an impediment in their rightful "content and delivery optimization techiques".

Things are getting even worse. Remember the concept of network neutrality? It is a very hot topic today and a very important political and economic fight is being fought right now to protect the transparency of the Internet. However, if you look further, you see that nobody considers the practice of "optimization" as something net neutrality should protect. In December 2010, the US Federal Communications Commission set three basic rules for net neutrality:
  • Transparency. Fixed and mobile broadband providers must disclose the network management practices, performance characteristics, and terms and conditions of their broadband services;
  • No blocking. Fixed broadband providers may not block lawful content, applications, services, or non-harmful devices; mobile broadband providers may not block lawful websites, or block applications that compete with their voice or video telephony services; and
  • No unreasonable discrimination. Fixed broadband providers may not unreasonably discriminate in transmitting lawful network traffic.
So they are primarily focusing on blocking and throttling, but not on preserving the integrity of the transmitted data!