Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Microsoft Plays Dirty With Gmail

By riddermark in Technology
Sun Jun 15, 2008 at 03:17:34 PM EST
Tags: microsoft, ie7, gmail, google (all tags)

A friend sent out the latest clip for Ghost Humpers, our episodic mockumentary. Downloading the quicktime movie from Gmail gave me a compressed version. I thought that was odd. After a little searching, I found an option to download attachments as a zip file. Simply replace "disp=attd" with "disp=zip" in the attachment URL. IE7 was changing this to "disp=indzip" for the same result; on its own. This piqued my curiosity.


ADVERTISEMENT
Sponsor: rusty
This space intentionally left blank
...because it's waiting for your ad. So why are you still reading this? Come on, get going. Read the story, and then get an ad. Alright stop it. I'm not going to say anything else. Now you're just being silly. STOP LOOKING AT ME! I'm done!
comments (24)
active | buy ad
ADVERTISEMENT
I searched Gmail for more attachments.  A WMV file gave me an unmodified URL. A DOC file gave me an unmodified URL. Files with Apple specific extensions of MOV, MOVIE, MOOV, MOVIEPROJ, QT, and QTCH gave the modified URL. What was going on?

Firefox, Safari, and Opera all gave me the unmodified URLs for every file. Therefore, there are only two options: either Google is giving IE7 a modified URL or Microsoft has coded IE7 to look for a list of extensions within Gmail and modify the attachment URL. The first option seemed highly unlikely.

The result of what Microsoft had done was obvious. A stumbling block stood in the way of getting to files downloaded through Gmail's webmail. This stumbling block required unzipping the file and it only appeared with Apple specific files. Why would Microsoft go to all the trouble of coding IE7 to make it harder for Gmail users to get to files that weren't from Microsoft? And what other kinds of files was Microsoft's product coded to look for? There was one way I could find out.

By copying every page of extensions from filext.com to a text file called extensions.txt, I had a list of every extension they are aware of. How many extensions were in the list?


wc -l extensions.txt

This showed 24656 lines. The file contained a bunch of superfluous information on every line, so I removed everything except the extension itself:

sed -i 's/ .*//' extensions.txt

The file also contained many duplicate file extensions.

cat extensions.txt | uniq > ext.txt

The file ext.txt contained 14037 lines. Now to create 14037 files:

cat ext.txt | while read line; do echo "${line}.${line}" > "${line}.${line}"; done

Using Evolution and Gmail's smtp server, I sent myself all these files in batches of about 100. The first and last extension of each batch was written down to keep track of my progess. When a batch was rejected, this was noted. Of 137 batches sent, 27 were rejected for containing extensions banned by Google.

By dividing up each rejected batch into smaller and smaller chunks, a list of 33 banned extensions emerged:


ADE
ADP
BAT
CHM
CMD
COM
CPL
EXE
HTA
INS
ISP
JSE
LIB
MDE
MSC
MSP
MST
PIF
SCR
SCT
SHB
SYS
TAR
TAZ
TGZ
VB
VBE
VBS
VXD
WSC
WSF
WSH
ZIP


From within Gmail's webmail, using IE7, I opened each batch and slid the cursor down every attachment, while watching the corresponding URL. It was immediately apparent that IE7 was programmed with an exception list, as every URL was modified. There were 31 exceptions:

DOC
GIF
GIF2
GIF87
GIF89A
GIFA
GIFENX
GIFF
GG
ID3
JPE
JPEG
JPG
MP2
MP3
MPE
MPEG
MPG
MPGA
PDF
PNG
PPS
PPT
SGIF
TIF
TIFF
WAV
WAVEBNK
WMA
WMV
XLS


Google already has safeguards in place that ban executable files with an exclusion list, so Microsoft can't claim this mantle. They go to all the trouble of creating IE7 with code specific to Gmail, to give people zipped versions of files that aren't on their list. No other browser puts this petty stumbling block in people's way.

Note: IE7 only mucks with Gmail in standard mode.

-Caleb

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Google
o Ghost Humpers
o found
o filext.com
o Caleb
o Also by riddermark


Display: Sort:
Microsoft Plays Dirty With Gmail | 56 comments (54 topical, 2 editorial, 0 hidden)
+1, Front Page. Keep up the good work gumshoe. (2.20 / 5) (#1)
by Ruston Rustov on Sat Jun 14, 2008 at 02:58:00 AM EST


I had had incurable open sores all over my feet for sixteen years. The doctors were powerless to do anything about it. I told my psychiatrist that they were psychosomatic Stigmata - the Stigmata are the wounds Jesus suffered when he was nailed to the cross. Three days later all my sores were gone. -- Michael Crawford
Maybe tomorrow. -- Michael Crawford
As soon as she has her first period, fuck your daughter. -- localroger

So GMail bans zip files? (2.50 / 2) (#2)
by it certainly is on Sat Jun 14, 2008 at 05:49:37 AM EST

You've said that GMail rejects attachments with the .ZIP extension. So .ZIP files must somehow be wrong , and GMail not like them.

But you also say that GMail presents things in the .ZIP extension on demand. So they must like it. Which is it?

Perhaps GMail rejects invalid ZIP files, rather than just because of their extension?

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.

Gmail and Zip Files (none / 0) (#3)
by riddermark on Sat Jun 14, 2008 at 06:20:59 AM EST

Gmail has an option to download any attachment as a zip file. However they ban attachments with extensions of .zip.

[ Parent ]
How utterly schitzophrenic (none / 0) (#4)
by it certainly is on Sat Jun 14, 2008 at 07:01:32 AM EST

They recognise the utility of zip files in bundling up files, but won't let you use them yourself.

(I don't refer to schitzophrenia, of course. I refer to multiple personality disorder, which is synonymous with the popular conception of schitzophrenia)

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.
[ Parent ]

This is obviously a post... (none / 0) (#5)
by mirleid on Sat Jun 14, 2008 at 09:25:43 AM EST

...by the other one of you who can't spell...

Chickens don't give milk
[ Parent ]
last time I checked (none / 0) (#44)
by vqp on Fri Jun 20, 2008 at 09:58:02 AM EST

 gmail allows uploading zip files as long as they do not contain exe files inside. The workaround is to rename the file to .zi_ or whatever you want

happiness = d(Reality - Expectations) / dt

[ Parent ]
The author is wrong (surprise) (none / 0) (#9)
by curien on Sat Jun 14, 2008 at 12:18:23 PM EST

Gmail doesn't ban zip files.

The error message indicates that they ban zip files which (appear to) contain executables, but I think you're right about them also banning invalid zip files.

--
Murder your babies. -- R Mutt
[ Parent ]

Re: Invalid Zip Files (none / 0) (#11)
by riddermark on Sat Jun 14, 2008 at 01:47:38 PM EST

Good point. Gmail probably rejected the text file zip.zip with contents of zip for being an invalid zip file.

Gmail help on Zip files:
http://mail.google.com/support/bin/answer.py?hl=en&answer=9481

[ Parent ]

ZIP, TAR, TAZ, TGZ (none / 0) (#12)
by riddermark on Sat Jun 14, 2008 at 02:40:41 PM EST

Curien's point also applies to TAR, TAZ and TGZ.

[ Parent ]
NIP, JAP, WOP, NIG, PAKI, FAG, (3.00 / 5) (#13)
by EMOTIVE GUY on Sat Jun 14, 2008 at 05:00:27 PM EST

My point also applies to SPIC, DAEGO, CHINK, GOOK, ARAB, SKIV, and BRIT
_______________________________________________
They told me to go easy on cock for a few days, but I didn't listen
- MotorMachineMercenary

[ Parent ]
Wow, I can read slashdot archives on K5 now (2.71 / 7) (#6)
by GhostOfTiber on Sat Jun 14, 2008 at 09:53:00 AM EST

Your article is delicious.

[Nimey's] wife's ass is my cocksheath. - undermyne

Where's The Archived Story? (none / 1) (#27)
by riddermark on Sun Jun 15, 2008 at 11:29:23 PM EST

I have found no similar story on Slashdot. What archived story are you referring to?

[ Parent ]
I TOO HAVE WASTED HOURS DICKING AROUND ON THE COMP (2.71 / 7) (#7)
by debillitatus on Sat Jun 14, 2008 at 10:00:53 AM EST


Damn you and your daily doubles, you brigand!

Useless Use of Cat, -1 (2.57 / 7) (#8)
by alba on Sat Jun 14, 2008 at 10:42:24 AM EST

Corrections:

uniq extensions.txt > ext.txt
while read line; do echo "${line}.${line}" > "${line}.${line}"; done < ext.txt

This is a discgrace to geekdom. Burn this site down.



Useless temporary file (3.00 / 2) (#10)
by curien on Sat Jun 14, 2008 at 12:21:02 PM EST

$ uniq extensions.txt | while ...

--
Murder your babies. -- R Mutt
[ Parent ]
That file might not be sorted as uniq expects (3.00 / 2) (#19)
by BJH on Sun Jun 15, 2008 at 03:31:46 AM EST

$ sort extensions.txt | uniq | while ...
--
Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

[ Parent ]
On any modern Unix... (none / 1) (#22)
by bodza on Sun Jun 15, 2008 at 11:48:29 AM EST

sort -u extensions.txt | while ...
--
"Civilization will not attain to its perfection until the last stone from the last church falls on the last priest." - Émile Zola

[ Parent ]
Sorry, I use Solaris 8 /nt (none / 0) (#29)
by BJH on Mon Jun 16, 2008 at 07:13:32 AM EST


--
Roses are red, violets are blue.
I'm schizophrenic, and so am I.
-- Oscar Levant

[ Parent ]
What made you upgrade from Minix 4? (3.00 / 4) (#32)
by rpresser on Mon Jun 16, 2008 at 10:28:05 AM EST


------------
"In terms of both hyperbolic overreaching and eventual wrongness, the Permanent [Republican] Majority has set a new, and truly difficult to beat, standard." --rusty
[ Parent ]
Give me a break (3.00 / 2) (#37)
by FattMattP on Mon Jun 16, 2008 at 07:40:31 PM EST

You're hung up on removing cat for a one liner that will never be run again? What's the problem? Are you running out of CPU cycles and memory on your ZX81?

It's hardly useless. It lets him read from left to right what he wants the line to do. Sometimes it's easier to make sure you understand what you are doing, or express a problem in a way that is comfortable, rather than optimizing for efficiency. Who cares that the computer has to start another process. Human time and attention is far more valuable than a couple of CPU cycles.

[ Parent ]

BS excuse (3.00 / 3) (#38)
by curien on Mon Jun 16, 2008 at 09:03:04 PM EST

It lets him read from left to right what he wants the line to do.

My version with the pipe does just that and eliminates an unneeded temporary file. Secondly,
 $cat a | b
is the same as
 $<a b
which retains left-to-right order (though it does look kinda weird if you're not used to it).

Sometimes it's easier to make sure you understand what you are doing, or express a problem in a way that is comfortable, rather than optimizing for efficiency.

Now THAT's a good reason. Hell, I do 'cat x | y' sometimes myself. However, we expect articles to be proofread and revised with an attentive eye. The author's sloppiness with Unix commands is but one symptom of the article's pervasive lack of rigor.

--
Murder your babies. -- R Mutt
[ Parent ]

Proofreading (none / 0) (#39)
by riddermark on Tue Jun 17, 2008 at 02:41:56 AM EST

Writing something contrary to what was done is called misstating events. However, noticing "progess" and changing it to "progress" would qualify.

[ Parent ]
I'm sorry, We're Revoking Your Geek License (3.00 / 2) (#41)
by alba on Wed Jun 18, 2008 at 03:49:42 PM EST

Useless Use of Cat Award
[...] The fact that the same thread ("but but but, I think it's cleaner / nicer / not that much of a waste / my privelege to waste processes!") springs up virtually every time the Award is posted is also Ancient Usenet Tradition.



[ Parent ]
Inconclusive. (2.66 / 3) (#14)
by rpresser on Sat Jun 14, 2008 at 08:15:26 PM EST

You cannot be certain that Gmail is not changing its behavior based on the user-agent unless you use a network protocol analyzer to see what is actually being sent and received.  Clearly there is an unhealthy reaction going on, but assigning the blame is premature.
------------
"In terms of both hyperbolic overreaching and eventual wrongness, the Permanent [Republican] Majority has set a new, and truly difficult to beat, standard." --rusty
Google or Microsoft (none / 1) (#15)
by riddermark on Sat Jun 14, 2008 at 08:46:16 PM EST

I'm pretty sure Microsoft is to blame. But you're right. I can't be 100% sure. I'm about 99% sure, given Microsoft's history and the nature of what is happening. The addition of WAVEBNK to the exception list points to code within IE7, as the file is an "Unknown Apple II File (found on Golden Orchard Apple II CD Rom)." The fact that this is similar to another item on the exception list (WAV) tells me Microsoft can't even code their mucking correctly.

Another test to determine to remove my 1% doubt: use a browser that emulates the IE7 user agent.

[ Parent ]

Blame Microsoft (3.00 / 3) (#16)
by riddermark on Sat Jun 14, 2008 at 09:44:15 PM EST

I changed the user agent of Firefox to IE7 and tested the exclusion list. Google gives me unmodified URLs, so Microsoft code is clearly to blame.

In Firefox:

  1. open a new tab and enter about:config in the address bar
  2. right click and click "new string"
  3. enter general.useragent.override
  4. enter your IE7 user agent
  5. check your useragent at useragent.org


[ Parent ]
Not conclusive (3.00 / 2) (#17)
by curien on Sat Jun 14, 2008 at 10:57:14 PM EST

There are ways to do browser detection other than the user agent string. The only conclusive test is to actually analyze the network traffic.

Install Wireshark and see what's really going on.

--
Murder your babies. -- R Mutt
[ Parent ]

You Are Correct (3.00 / 2) (#18)
by riddermark on Sun Jun 15, 2008 at 12:44:57 AM EST

You're saying Google might be employing a method, other than the simplest method, to determine if the browser is IE7. The results of this method are then used to alter attachment download links, resulting in a stumbling block to files outside of the exclusion list.

You're right. They might be doing that. But the chances are so infinitesimally small, it is not worth pursuing.

[ Parent ]

Most complicated javascript systems... (none / 0) (#31)
by claes on Mon Jun 16, 2008 at 08:15:59 AM EST

try to do browser detection there, i.e. in javascript code.  Too many people muck with the user-agent string.

[ Parent ]
Javascript Detection (none / 0) (#43)
by riddermark on Wed Jun 18, 2008 at 07:58:14 PM EST

Javascript detection would give the same result, because it is derived from the same place.

[ Parent ]
That is not correct. (none / 0) (#50)
by ffrinch on Fri Jul 18, 2008 at 09:35:26 AM EST

People who know what they're doing use feature detection.

-◊-
"I learned the hard way that rock music ... is a powerful demonic force controlled by Satan." — Jack Chick
[ Parent ]
emulates is easy (none / 0) (#24)
by jettero on Sun Jun 15, 2008 at 04:24:16 PM EST

Simply grab User Agent Switcher.

I use it to log into my university class registration site (ugh).

[ Parent ]

user agent ie7/vista (none / 0) (#33)
by jettero on Mon Jun 16, 2008 at 02:37:42 PM EST

I actually did try this with a vista ie7 useragent string and it did not adulterate the download. I still find it highly unlikely ie7 is re-writing the url, but I'm more convinced than before.

[ Parent ]
Spot on. Wireshark does the trick. /nt (none / 0) (#30)
by claes on Mon Jun 16, 2008 at 08:14:28 AM EST



[ Parent ]
uh, so (2.50 / 2) (#20)
by Linux or FreeBSD on Sun Jun 15, 2008 at 07:25:05 AM EST

why don't you fire up a debugger and actually find the code that's doing this.

There's an easy way to figure out who's to blame (none / 1) (#25)
by xcham on Sun Jun 15, 2008 at 05:32:42 PM EST

Grab user agent switcher for Firefox and make it masquerade as IE. If you get the weirdo URLs in Firefox-pretending-to-be-IE, it's Google. If not, it's MS. THEN it's time to whip out a debugger.


Resolved in "Inconclusive" Thread (none / 0) (#26)
by riddermark on Sun Jun 15, 2008 at 06:13:27 PM EST



[ Parent ]
Addendum (none / 1) (#28)
by riddermark on Mon Jun 16, 2008 at 02:02:01 AM EST

Google's list of banned extensions has no bearing on the article's premise. However, it should be noted that TAR, TAZ, TGZ, and ZIP are banned if they are invalid files or contain executables.

The article left open the "highly unlikely" possibility of Google being responsible. This was reduced to "infinitesimally small" by changing the user agent of Firefox to IE7 and testing.

Can't reproduce (none / 1) (#34)
by ghjm on Mon Jun 16, 2008 at 03:47:40 PM EST

I was going to run Wireshark and prove you wrong, but Gmail and IE7 don't do this on my machine.

If you feel like it you can provide specific steps to reproduce and I'll show the error of your ways, but I'm just as happy for you not to bother.

kthx

Oh I get it now (3.00 / 4) (#35)
by ghjm on Mon Jun 16, 2008 at 03:55:05 PM EST

You have the Quicktime IE7 extension installed. Next topic please.

[ Parent ]
What Extensions Did You Test? (none / 0) (#36)
by riddermark on Mon Jun 16, 2008 at 07:15:32 PM EST

As the article clearly states, IE7 only modifies the download URL for files outside of the exception list.

[ Parent ]
Tried it with several (3.00 / 2) (#40)
by ghjm on Wed Jun 18, 2008 at 01:13:07 AM EST

Both on and off your list. mov, wmv, doc, zip, mp3, aac, some others. No extensions were modified. Then I enabled QTPlugin.ocx and got the behavior you describe. So on my machine at least, it looks like it is neither Microsoft nor Google - it's Apple.

[ Parent ]
Read It Again (none / 0) (#42)
by riddermark on Wed Jun 18, 2008 at 07:45:02 PM EST

The question you should be asking is which of the six within your test batch download as zip files? Unless you have a unique version of IE7, your answer will be "all except DOC and WMV."

[ Parent ]
NONE DOWNLOAD AS ZIP FILES (none / 0) (#45)
by rpresser on Fri Jun 20, 2008 at 12:15:38 PM EST

I don't have the quicktime plugin installed. Every file downloads exactly as desired, not as a zip file.  The URL is left strictly alone.

My URLs by default have an disp=safe in the middle of them, not disp=att:

http://mail.google.com/mail/?ui=2&ik=f921388b07&attid=0.1&disp=safe& amp;view=att&th=11aa6bd104092f3f

This downloads my mov file without interference.

If I change the safe to att, I get the same behavior: a perfect download, no interference.

If I change safe to zip, I get the zip file you described. Interesting trick, thanks for the tip.

But your paranoia about IE7 is unwarranted. If, as the other person said, the Quicktime plugin is responsible, then Apple is responsible, not Microsoft.
------------
"In terms of both hyperbolic overreaching and eventual wrongness, the Permanent [Republican] Majority has set a new, and truly difficult to beat, standard." --rusty
[ Parent ]

Test Environment (none / 0) (#46)
by riddermark on Sat Jun 21, 2008 at 02:16:56 AM EST

This entire thread might have been avoided if my IE7 test environment was mentioned: a freshly installed Windows system without any additional software installed.

[ Parent ]
The problem is not bothering us (none / 0) (#48)
by rpresser on Mon Jun 23, 2008 at 09:34:24 AM EST

in our current environments. We've done fairly thorough checks that it is not bothering us. That it is bothering you is no longer our problem.
------------
"In terms of both hyperbolic overreaching and eventual wrongness, the Permanent [Republican] Majority has set a new, and truly difficult to beat, standard." --rusty
[ Parent ]
Addendum II (2.50 / 2) (#47)
by riddermark on Sat Jun 21, 2008 at 02:45:43 AM EST

The IE7 test environment was a freshly installed and updated Windows system; without any additional software installed.

WOW, Great Work! (2.50 / 2) (#49)
by k31 on Tue Jul 01, 2008 at 01:41:37 PM EST

I would have +FP'd this if it wasn't already published... actual scientific investigation rather than just jumping to conclusions. Makes me have faith that humanity isn't just the viral plague upon the earth that Mr. Smith said it was.

Anyhow, yeah, Microsoft does play really dirty. Bad Imperialisic Overlord-z Company!

Your dollar is you only Word, the wrath of it your only fear. He who has an EAR to hear....

CONTROVERSY (3.00 / 2) (#51)
by TDS on Thu Jul 24, 2008 at 04:27:34 PM EST

AN ARTICLE on tech site Kuro5hin claims that the Internet Explorer 7 web browser fiddles with downloads from GMail by altering the download URLs and requesting zipped versions of files in Apple's file formats like Quicktime.

Caleb from Austin, TX who goes by the nickname riddlemark on the tech site recently wrote an intriguing piece entitled "Microsoft Plays Dirty With Gmail" describing his surprise to see that IE7 always downloaded .mov files sent as GMail attachments as compressed .zip files, whereas clicking on the same download link in Firefox and other browers gave him the .mov file straight from the cooker without a chaser.

...coverage here.

LOL, they referred to Kuro5hin as a "tech site".

And when we die, we will die with our hands unbound. This is why we fight.

Re: Controversy (none / 0) (#52)
by riddermark on Tue Sep 02, 2008 at 01:14:06 AM EST

While investigating the validity of the author's story, I found the code that alters the URL was introduced at least as early as IE6. IE7, running on Windows XP, continues to alter the download URL to this day (version 7.0.5730.13).

Microsoft's code is also flawed. My original test showed GIF2, GIF87, GIF89A, GIFA, GIFENX, GIFF, ID3, SGIF and WAVEBNK to be on the exception list. Viewing these attachments within their original grouping of about 100 still shows this. However, when these exceptions are viewed as part of a message containing all 31 attachment exceptions, they no longer appear as exceptions. In other word, IE6 or IE7 doesn't make an exception for these attachments, and alters the download URL.

[ Parent ]

Addendum III (none / 0) (#53)
by riddermark on Sat Sep 20, 2008 at 04:08:12 AM EST

The code that alters the URL was introduced at least as early as IE6. IE7, running on Windows XP, continues to alter the download URL to this day.

Microsoft's code is also flawed. My original test showed GIF2, GIF87, GIF89A, GIFA, GIFENX, GIFF, ID3, SGIF and WAVEBNK to be on the exception list. Viewing these attachments within their original grouping of about 100 still shows this. However, when these exceptions are viewed as part of a message containing all 31 attachment exceptions, they no longer appear as exceptions.

People discovered this behavior at least as early as April 28th, 2006.

(IE7 on Vista behaves this way as well.) (none / 0) (#54)
by riddermark on Sat Sep 20, 2008 at 04:24:26 AM EST



[ Parent ]
Addendum IV (none / 0) (#55)
by riddermark on Sat Sep 20, 2008 at 05:59:32 AM EST

The extension TXT is on the exception list. Faulty Microsoft code continues to alter its download URL within the original message that contains around 100 attachments.

Microsoft yahoo and google (none / 0) (#56)
by chrisranjana on Thu Dec 04, 2008 at 08:11:36 AM EST

Why there are not many players ?

only those 3 ?

What happens if they join together?

--------
chris,Director, Chrisranjana Software and Solutions PVT LTD,
Chennai, Tamil Nadu, India,
Custom php Mysql programming and Internet web development

Microsoft Plays Dirty With Gmail | 56 comments (54 topical, 2 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!