Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Mistaken Disclosure

By Miniluv in Op-Ed
Wed Jun 26, 2002 at 08:02:13 PM EST
Tags: Security (all tags)
Security

The past few weeks have seen a couple widely used servers, namely OpenSSH and Apache HTTPD as the subject of security vulnerability announcements. There were some interesting contrasts between how the two disclosures were handled, which is what I intend to examine here.


First we have the Apache hole with their disclosure regarding it. The vulnerability was discovered by ISS who also issued a patch of their own. It was quickly discovered that this patch did not work, and Apache issued an official patch, along with new versions of their two supported branches. Apparently the Apache group took ISS's word for it when they said that it was only exploitable with arbitrary code execution on 64-Bit platforms, such as Solaris on Sparc64, as opposed to the 32-Bit platforms, such as Linux on IA32, which turned out to be wrong as well. Gobbles posted a working exploit against OpenBSD running Apache on Intel hardware. The Apache group responded with an update to their advisory stating that the already released patches protected against the remote execution of code, in addition to the previously known denial-of-service(DoS) attack.

Contrast this with the much more recent OpenSSH vulnerability adventure which began with a post to the openssh developement list which quoted a post to the BugTraq security mailing list. Theo de Raadt of OpenBSD fame made statements to the effect that if people didn't upgrade to the current release of OpenSSH, at the time 3.3, and enable privilege separation there were going to be big problems later this week when a full vulnerability announcement was made, in addition to a release with the hole fixed. Beyond this Theo also took the time to expound on the lack of support he was receiving from the Linux community, HP and other vendors who felt that privilege separation was a dubious concept with no proof of real world benefit. Today things changed when the ISS announcement and then OpenSSH disclosure hit the web. At the same time OpenSSH 3.4 was released, which closes the hole. What's different about this from every other security vulnerability you ask? The presentation of course. Theo made this vulnerability sound widespread and serious, most likely a remote root hole (which it is) that affected most, if not all, users of OpenSSH (which it doesn't). Vendors rushed packages out the door to protect their users, despite the lack of information flowing their way, only to find out that the default configurations they were shipping were not vulnerable.

Upon further scrutiny, it appears the only systems vulnerable by default were OpenBSD, FreeBSD-Current, and possibly NetBSD. This raises a serious question regarding Theo's behavior in the initial disclosure on BugTraq. Why did he hammer everyone so hard to upgrade? Why was he pushing so hard for people to use a new and still somewhat experimental feature which lacks proper support on many of the "portability" platforms which OpenSSH supports?

I think these two sets of circumstance highlight the dangers vendors and security analysts face when making announcements. The dangers are sounding too serious, a la OpenSSH, or underestimating the problem as happened with Apache. While it's important to disclose all of the facts, and not whitewash the potential threat of any given hole, it's equally important not to panic your userbase and end up sounding like a cross between chicken little and the boy who cried wolf. Ultimately I think that OpenSSH crew should have released a fix much sooner, it was a trivial couple lines to add to prevent that integer overflow. I think they should've released more details rather than forcing a large number of users to hose their servers when all of the sudden people realized "Whoops, PAM and PrivSep don't play well together on Linux". I think Theo needs to decide whether he actually cares about his users, or if he cares purely about OpenBSD support. If the latter then stop releasing the portable version and let the community fork it into something that actually makes a solid attempt at cross platform support. Then we can rip out flaky and dubious features like privilege separation and any other similar schemes that might be dreamed up.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Did Theo screw the pooch?
o Yeah, and forgot the lube too! 43%
o Maybe a bit 25%
o Only by accident 3%
o Not really 16%
o Absolutely not 10%

Votes: 55
Results | Other Polls

Related Links
o OpenSSH
o Apache HTTPD
o hole
o disclosure
o ISS
o posted
o a post
o OpenBSD
o ISS announcement
o OpenSSH disclosure
o Also by Miniluv


Display: Sort:
Mistaken Disclosure | 59 comments (49 topical, 10 editorial, 0 hidden)
OpenSSH 3.3 had problems with Linux 2.2 (4.50 / 2) (#4)
by tftp on Wed Jun 26, 2002 at 01:55:28 PM EST

when all of the sudden people realized "Whoops, PAM and PrivSep don't play well together on Linux"

Don't know about that, but what I definitely do know is that on Linux 2.2 mmap() works differently than on 2.4, and as result sshd crashed on connect attempt. I had to search for a patch, and then to apply it by hand, and then recompile and manually install the affected executable.

To summarize, OpenSSH 3.3 -as shipped- would be crashing endlessly on any Linux 2.2 box with a vague error message "mmap() failed". That is an instant DoS. I pity sysadmins who dared to upgrade older OpenSSH without trying the new one on a test box. One needs to have a working knowledge of C programming to fix this problem. Or maybe OpenSSH 3.3 should not have been pushed down our throats before it was well tested on all supported platforms.

Hm (none / 0) (#15)
by lb008d on Wed Jun 26, 2002 at 02:29:18 PM EST

I pity sysadmins who dared to upgrade older OpenSSH without trying the new one on a test box.

And I pity employers who hired that sysadmin who's too stupid to test something as critical as a new version of SSH before putting it into production.

OpenSSH 3.3 should not have been pushed down our throats before it was well tested on all supported platforms.

Who's pushing it down your throat? You can disable the Challenge/Response protocol and had you waited a day for Theo to announce that you wouldn't have had to upgrade.

The best response to this for most sysadmins would have been to kill sshd and wait until the smoke clears. In mission-critical situations you could have upgraded, as you did. Is it really that hard to apply a patch by hand (as opposed to what, downloading an rpm?) and re-compile the program?

He delayed that announcement until a definitive fix could be put in - he didn't want to give ammo to anyone working on an exploit.

[ Parent ]

Hmm (4.00 / 2) (#18)
by tftp on Wed Jun 26, 2002 at 02:45:11 PM EST

And I pity employers who hired that sysadmin who's too stupid to test something as critical as a new version of SSH before putting it into production.

It is sometimes difficult to have test boxes for every production server, with exactly the OS and the kernel and the environment identically set... Of course, one could test the sshd on the production server simply using a different port.

You can disable the Challenge/Response protocol and had you waited a day for Theo to announce that you wouldn't have had to upgrade.

It is equally difficult (or equally easy) to find out what to patch as to find out what to disable. I was not worried, I could go back to previous build at any time.

The best response to this for most sysadmins would have been to kill sshd and wait until the smoke clears.

I had to keep sshd running because people used it and needed it.

Is it really that hard to apply a patch by hand (as opposed to what, downloading an rpm?) and re-compile the program?

For me it is not difficult (I read diffs all day long); for some people it may be a problem. With regard to the announcement, it is clear that 3.3 hasn't had a chance yet to be tested well (it was released less than a week before). The announcement forced many people to upgrade to a new software with new, never tried before features, even though most had no need to do that. It was somewhat misleading, maybe even reckless.

[ Parent ]

Forced? (5.00 / 1) (#22)
by lb008d on Wed Jun 26, 2002 at 02:55:58 PM EST

The announcement forced many people to upgrade to a new software with new, never tried before features, even though most had no need to do that.

Let me re-iterate: Announcements don't force anyone to do anything. They give advice.

Smart syadmins weigh risks and make decisions based on those risks. In your case, you had to weigh your users needs, the possible exploit at hand, and the newness of the software being presented as a proposed fix. You did the right thing by testing it first. Any smart sysadmin should have looked at the newness of 3.3 and hesitated to install it before testing. They should have weighed measures to block the exploit, even if it meant users had to go 4 hours without ssh while a test was performed, or if it meant leaving the box exposed for that amount of time while testing.

Smart sysadmins also test their software before installing it. As you said, you don't need an identical box to test something before putting it in production. The mmap error you described could have easily been reproduced on any 2.2 box.

[ Parent ]

The problem (3.00 / 1) (#25)
by Miniluv on Wed Jun 26, 2002 at 03:06:32 PM EST

This is where many of us have an issue with the tone and content of the announcement Theo made. He made it sound like the world was going to end if we weren't upgraded by the exact moment 3.4 came out. This was disingenous at best, dangerous at worst.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]
Your problem, not his (5.00 / 1) (#26)
by lb008d on Wed Jun 26, 2002 at 03:18:18 PM EST

He made it sound like the world was going to end if we weren't upgraded by the exact moment 3.4 came out

I see your hyperbole, however, anyone who thinks the "world will end" due to not upgrading to new software becuase of an undisclosed, less-than-a-week-old exploit shouldn't be a sysadmin. You've got to stop and think, and weigh risk.

I'll bet you'd be reacting with the complete opposite tone should Theo had not made this seem important and an exploit rooted your machine.

Sysadmins are 100% responsible for their machines and what happens to them. Laying the blame elsewhere is a cop-out.

[ Parent ]

Not really (none / 0) (#31)
by Miniluv on Wed Jun 26, 2002 at 04:00:48 PM EST

Absolutely it's my responsibility to maintain control of my servers, however I can only make decisions based on the information presented to me, and Theo did the community a huge disservice with the quality of information he disseminated.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]
Bingo! (none / 0) (#38)
by lb008d on Wed Jun 26, 2002 at 04:50:35 PM EST

I can only make decisions based on the information presented to me

Let's see what info a typical sysadmin would have when this is announced:

  1. Theo is working on an "upcoming OpenSSH vulnerability"
  2. When sshd is running with privsep, no expoit can occur
  3. OpenSSH 3.3 was released a few days ago and is not yet perfect
  4. Theo: Everyone should update to OpenSSH 3.3 immediately, and enable priv seperation in their ssh daemons
  5. No exploits are known
Based on that amount of info given, had I been administering a Linux system that may have PAM problems, I would have thought more than twice about upgrading, and would have considered shutting down ssh just to be sure to test the new version myself.

Yes, Theo used strong words, but strong words shouldn't force an admin into rash decisions.

[ Parent ]

True (none / 0) (#45)
by Miniluv on Wed Jun 26, 2002 at 05:26:37 PM EST

And that's what we did in my shop. That doesn't change the fact that Theo used words that I feel were stronger than the situation warranted. Nor does it excuse the fact that he made an announcement that, apparently, was merely for the sake of making an announcement.

Most professionals weren't affected by this whole flap, as we knew enough to sit back and wait a little bit, or to thoroughly test a solution to implement privsep if we had that option. A lot of newbies and less experienced admins locked themselves out of boxes, or spent a good bit of time biting their fingernails wondering if they were sitting ducks because Theo worded things strongly.

Most advisories try to paint an accurate picture, giving worst case analysis that is clearly worded as such. Release early, release often isn't a mantra for security disclosure, it's a release cycle. Security disclosures should follow a more simple principal of "release accurately".

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

I disagree (none / 0) (#55)
by tzanger on Thu Jun 27, 2002 at 10:31:21 AM EST

I'll bet you'd be reacting with the complete opposite tone should Theo had not made this seem important and an exploit rooted your machine.

No I wouldn't. Theo flew off the handle on this one trying to get new code shoved down everyone's throat. I've upgraded sshd many times (and all too many times recently it seems) based on normal, calm, properly written announcements. This seemed like a PR stunt.

For the record, I didn't even upgrade from 3.3; Adding "ChallengeResponseAuthentication no" (or uncommenting it) is all that needs to be done. There was absolutely no reason for Theo to fear-monger like he did.



[ Parent ]
I hate the constant upgrade advice (none / 0) (#52)
by 0xA on Thu Jun 27, 2002 at 04:49:15 AM EST

I'm not all that really excited about rushing in a 3.4 install either. It always bugs me when "Install this new version" is offered as a solution to something. I'd much rather patch and compile.

This is even more of a problem in the Windows world, you always ahve to wonder what crap is going to come along with when you upgrade something for whatever reason. You usually can't jsust patch and recompile. I was a little nervous when I read Theo's message earlier this week but now that I know what the problem is I'll be much less woried with CHAP turned off as opposed to wondering what's wrong with 3.4

I wasn't using CHAP anyway.

[ Parent ]

Or (none / 0) (#54)
by tzanger on Thu Jun 27, 2002 at 10:27:41 AM EST

It is sometimes difficult to have test boxes for every production server, with exactly the OS and the kernel and the environment identically set... Of course, one could test the sshd on the production server simply using a different port.

You could do that, or, as I often do: Save the old bin/libs "just in case" and then install the new version of ssh. Now type pstree -p | grep sshd and kill off the parent process -- your session won't die. Now you can try running the new version and logging in again (leaving the original window open) to see if things work. If they do, great. If not, you have the old binaries to put back (and reinstall from source if you want to make sure it's ALL back (man pages, etc.).



[ Parent ]
Simpler fix than patching (none / 0) (#29)
by Anonymous Commando on Wed Jun 26, 2002 at 03:39:55 PM EST

I've got OpenSSH 3.3 working on a RH 6.2 box (using the 2.2 kernel). I added Compression no to my sshd_config file, and it worked for me without any patching (configured with --with-pam --with-tcp-wrappers --with-ipv4-default).

And I can't code C worth crap (I'm a Perl guy myself) - I found the fix posted to /. when Theo first "pre-announced" the possible vulnerability.

Aw crap, I just mentioned "the other site" - kiss my mojo goodbye...
Corporate Jenga™: You take a blockhead from the bottom and you put him on top...
[ Parent ]

Feh (4.25 / 8) (#6)
by trhurler on Wed Jun 26, 2002 at 02:00:30 PM EST

Theo did the right thing. Remember, he is not responsible for and does not know everyone's default configurations, and this is a remote root hole. Further, these sorts of holes tend to be exploitable in more than one way, so even if s/key isn't enabled by default, if the code is in your binaries, it is possible someone could find a way(possibly using another hole in combination) to exploit it.

If everyone handled security as seriously as the OpenBSD guys do, while there would still be bugs and security problems, breaking into systems would be unheard of - it would just be too difficult to be worthwhile. That goal justifies a certain amount of hype.

--
'God dammit, your posts make me hard.' --LilDebbie

I disagree (4.00 / 3) (#7)
by Miniluv on Wed Jun 26, 2002 at 02:04:13 PM EST

Theo could have made a meaningful announcement, however that would've required that they not pussyfoot around a week with releasing a patch. That fact right there is one of my major irritation points with this whole situation.

Issuing broad warnings that a version has some sort of exploit is bad, telling people that the only way to fix it is to turn on a brand new feature rather than simply turning off a different feature is a bad, bad precedent.

This is the same sort of enforced upgrade with enforced feature creep that people clobber Microsoft about all the time, yet it's ok when the bastion of Open Source Security does it? Somehow that just doesn't fly with me.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

I agree with your point here (none / 0) (#13)
by notafurry on Wed Jun 26, 2002 at 02:10:19 PM EST

At the same time, you didn't say that in the article and it really does make a big difference.

[ Parent ]
Think of it differently (5.00 / 2) (#14)
by lb008d on Wed Jun 26, 2002 at 02:14:38 PM EST

Issuing broad warnings that a version has some sort of exploit is bad, telling people that the only way to fix it is to turn on a brand new feature rather than simply turning off a different feature is a bad, bad precedent.

The fix would be to turn off ChallengeResponseAuthentication in the sshd_config file. Had Theo announced that, it would have narrowed the search for an exploit to a much smaller section of code than just saying "There's a problem, upgrade to fix it."

He didn't give any ammo to those who may be working on an exploit as we speak, and gave people a fix that, while being more of an inconvienience, will benefit them in the end anyway.

Go learn a little more about OpenSSH's "feature creep" before bashing it.

[ Parent ]

Nonsense (5.00 / 1) (#23)
by rantweasel on Wed Jun 26, 2002 at 03:03:38 PM EST

I think it makes sense to do things the way the OpenSSH crew did.  Lots of vendors distribute OpenSSH, and I think giving fair warning to them that there is something major coming up, and they need to set aside some time for QA and compatiblity issues makes sense.  It's certainly less problematic than Apple's response (none so far to either the Apache or OpenSSH bugs), or the Debian announcement about the OpenSSH bug (whiny and bitchy, if you don't like the authors of OpenSSH and the way they handle their software, write DebSSH) or others like them.  It's not an enforced upgrade or feature creep, it's common sense and fair warning.  Everybody knows that avoiding running things as root that don't need to be run as root is a basic security measure.  There's a www user, a pop3 user, a nobodry user, etc.  How is that any different from OpenSSH using the sshd user where it can?  What feature creep is involved?  It's basic security that's been applied to ftpd and httpd for years.  As for the enforced upgrade, the email that I saw said something along the ideas of "It's a really good idea if..."  and "Don't say we didn't warn you..." but nobody held a gun to my head.

mathias

[ Parent ]

Oh come on (none / 0) (#32)
by Miniluv on Wed Jun 26, 2002 at 04:11:08 PM EST

Lots of vendors distribute OpenSSH, and I think giving fair warning to them that there is something major coming up, and they need to set aside some time for QA and compatiblity issues makes sense.
And you think this disclosure actually did that? Do you honestly believe that any vendors benefitted from the manner in which this was handled? I think Debians response was entirely appropriate and that they raised very good points. Typically when software vendors coordinate with distros there's actual information flow, as in "here's the bug, here's the upcoming fix, you have until day X to get your packages ready" not "there's a bug, we're not telling you what, the release to fix this comes out sometime in the next week or so."

It's not an enforced upgrade or feature creep, it's common sense and fair warning.
So tell me, do you still consider yourself on "high alert" thanks to the fair warning Tom Ridge has given America oh-so-many times? And yes, it is an enforced upgrade in the sense that Theo gave no one any options for being ready for the full details coming out other than using a brand new feature that many people are still evaluating the usefulness of. Yes yes, there's the overused excuse that "more details would've given away the exploit" however it was pretty apparent what portion of the app was affected just by what he said. I'm no exploit coder and I knew it had to be a pre-authentication overflow, because that's about all that privsep protects you from.

It's basic security that's been applied to ftpd and httpd for years.
Both of those still launch as root, then spawn children which drop privileges. OpenSSH has decided that a better way is to launch as root, spawn an unprivileged child, then add overhead by using pipes to communicate between the child and parent. This doesn't make a heck of a lot of sense to me, as the ftpd and httpd way has been tested thoroughly, the sshd way is brand new.

As for the enforced upgrade, the email that I saw said something along the ideas of "It's a really good idea if..." and "Don't say we didn't warn you..." but nobody held a gun to my head.
What the email was guilty of was hyperbole and irrelevancy cloaking the actual issue of the security hole. This is classic Theo, who refuses to be open and honest with his users in a productive manner. For the people who worship him for it, that's fine. Allow him to cloak himself in this aura of devotion to quality code and forget that he has none of the good qualities of a strong project leader.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]
Not entirely unreasonable (none / 0) (#48)
by rantweasel on Wed Jun 26, 2002 at 06:03:56 PM EST

And you think this disclosure actually did that? Do you honestly believe that any vendors benefitted from the manner in which this was handled?

Well, the OpenBSD authors can't control how the vendor responds, they can only let them know what's going on.  I don't know how much information they provided to vendors, or how they provided it, or what vendors, etc.  I do know the advisories I saw (or didn't see), and I know that any of the vendors who aren't happy with OpenSSH can either fork the code or start their own version, so they have little right to bitch that they didn't like the advance warning that they got.  Ditto for implementation of the privilege separation - if you don't like the way it's done, fork.  Or use a different sshd.

As for the level of detail, sure, you can figure out that it is something pre-auth, but that is less specific than knowing which 500 lines of code the flaw is in.  I'm all in favor of full disclosure, but in the case of a vendor saying that "we've found a bug, we're releasing a patch, in the meantime do -something-", I'd rather that they wait on the the disclosure until the patch is out.

I agree with you to some extent, there was a lot of hyperbole, but I'm not entirely sure that we've seen the full extent of the hole, either.  If you look at the 2nd openssh.org advisory (from Markus Friedl) on the topic, there's a second hole in PAM auth, which hasn't been discussed much.

mathias

[ Parent ]

So then... (5.00 / 2) (#30)
by trhurler on Wed Jun 26, 2002 at 03:46:52 PM EST

Given the feature we're talking about(priv sep,) I have a question for you.

If you were using a system with no passwords, and I said "look, I am aware of this hole, but you'll have to enable passwords to get around it" would you be pissed at me for that?

If so, are you stupid?

Privelege separation is an obvious and easy security improvement. It is not a "feature" in the sense in which security people berate feature creep. If you can't see the difference, then the problem is that you don't know what you're talking about.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Why I dislike privsep (3.00 / 1) (#33)
by Miniluv on Wed Jun 26, 2002 at 04:17:43 PM EST

The concept sounds pretty nifty doesn't it? Run the minimal amount of code in a privileged mode, and run the rest in userspace, chrooted, etc. However, why add the complexity of shared memory and pipes when you could just drop privs after authentication? No matter what you have to authenticate a password as root, unless you care to be dumb enough to allow others to read /etc/shadow.

Beyond that is the fact that they've released it just recently, it's shaky at best on a well deployed platform, and the only excuse they have for that is that Alan Cox doesn't like Theo. We'll not really get into the "not compatible with 2.2.x mmap" and the numerous PAM issues.

Finally, your analogy is asinine. I would object vehemently if passwords were subtly incompatible with my current operating system, and required custom patches to be found in random spots on the internet. I would object vehemently if the concept of a password was only publicly released last week, and then suddenly you're telling me its the only salvation of my machine.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

So... (5.00 / 2) (#36)
by trhurler on Wed Jun 26, 2002 at 04:33:02 PM EST

Let me make sure you are following this: yes, there are shared memory and pipes. Nevertheless, the amount of code that ever runs with privelege drops to roughly 10% of what it once was.

I, one man, can thoroughly audit the less than 3000 lines that now "matter." When it was closer to 30,000, nobody could do it; the mere rate of change in the codebase would ensure that a whole army would find it impossible.

In any case, anyone who waited to hear the vulnerability announcement also got 3.4, which fixes the bug even without priv sep. Anyone who wanted a fix before that had one. I think the major problem here is that you have a problem with Theo, rather than any real practical problem.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Not really (none / 0) (#43)
by Miniluv on Wed Jun 26, 2002 at 05:23:20 PM EST

If not even an army could audit 30K lines of code, how on earth can OpenBSD claim thorough security audits?

As for my problem with Theo, yes, I admit I have one. I dislike the way he deals with his userbase, and that was only reinforced when I went back and read the email archives from the netbsd "incident". That's beside the point here though, as I also respect Theo a lot for his efforts with OpenBSD, and I generally admire the OpenSSH project. I just have a problem with the way this particular incident was handled.

I don't like people declaiming that the sky is falling when it's not, and giving vague nebulous warnings and then contradicting themselves rather quickly.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

30k (5.00 / 1) (#47)
by trhurler on Wed Jun 26, 2002 at 05:57:28 PM EST

It isn't about the absolute size so much as the change rate and number of relatively permanent dedicated developers. Most of OpenBSD's code doesn't change except for bug fixes, and there are a LOT of dedicated people who work on it, whereas OpenSSH, even if you had a huge audit team for a month or so, is only ever going to have a handful of developers, and after a month, the audit team still won't know what the hell is going on. Or a year. Or any time period. Too much change, too much code. That's why priv sep is a good idea.

Anyway, Theo doesn't care. He's there to produce good code. He feels some responsibility not to put people at undue risk, and some responsibilty to maintain the code quality and free license of BSD, but other than that, I don't think he gives a fuck about users, except as a means of income to fund the project. Why should he? They don't care about him, and they only care about his product in a "how can I make use of this" way. Most of them are too cheap to even buy a CD. I'd say fuck em too. Theo's right:)

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Yes.. (none / 0) (#58)
by mindstrm on Fri Jun 28, 2002 at 12:26:43 AM EST

It is not necessary to fix the problem though.

That's the problem.

Theo could have told everyone how to fix the vulenrabiliy rather than using it as a platform to launch his own changes.


[ Parent ]

hey, cool (5.00 / 3) (#8)
by tps12 on Wed Jun 26, 2002 at 02:07:05 PM EST

I just checked the OpenBSD web site, and they've changed the text to read "1 remote root exploit." How long has it been like that?

Probably (5.00 / 1) (#11)
by lb008d on Wed Jun 26, 2002 at 02:09:39 PM EST

just today or yesterday. It's nice they're keeping up on that, even when no working exploits are available yet (AFAIK).

If you ask me it's still an impressive track record on security.

[ Parent ]

Get a clue (4.60 / 5) (#10)
by lb008d on Wed Jun 26, 2002 at 02:08:16 PM EST

...stop releasing the portable version and let the community fork it into something that actually makes a solid attempt at cross platform support

You do understand the BSD license, right? There's nothing stopping anyone from taking OpenSSH and doing whatever they want with it.

In fact it's the OpenSSH team's ability and graciousness to produce a portable version that gives people the opportunity to have a free SSH server.

rip out flaky and dubious features like privilege separation

Go right ahead, but had you done any research into privsep you'd realize it's a good idea. The fact that it doesn't work in certain situations isn't helped by the lack of support from other vendors either. If they want a free SSH to work on their systems, then by all means take it and run.

I do know the OpenSSH team has more than proven their ability to write an outstanding product, however.

Read again (4.00 / 1) (#16)
by Miniluv on Wed Jun 26, 2002 at 02:32:28 PM EST

Yes, the BSD license allows it, hell so does the GPL. The problem is that people use portability code that has not always lived up to the standards that the OpenBSD code has. This is not necessarily a bad thing for OpenSSH, but it is for the userbase which, I suspect, is much larger for the portable versions than the OpenBSD version.

I've read a fair bit about privilege separation, and I see it being of dubious merit at best. The main gain is that you don't get a root exploit every time, however it's still possible. You're moving the input validation code into the client, however the data still has to go to the privileged parent process, and thus you can still exploit the box. Yes it's harder, not it's not impossible. In fact, I doubt it's even that much harder. If you can make the unprivileged process execute arbitrary code, then why can't it turn around and overflow a buffer in the parent?

I agree in general that OpenSSH is an outstanding product. I have problems with them following Theo's less than outstanding example of shoving his views down users throats, rather than showing a little maturity and responsibility and perhaps acknowledging their needs.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

Simplicity (5.00 / 1) (#17)
by lb008d on Wed Jun 26, 2002 at 02:41:36 PM EST

It all comes down to this: the fewer lines of privliged code, the easier it is to ensure they're bug-free.

[ Parent ]
Hmmm... (4.00 / 1) (#21)
by Miniluv on Wed Jun 26, 2002 at 02:54:53 PM EST

The entire OpenSSH project is a whopping 27K lines. Obviously the 2500 of which are running as root under privsep are easier to secure than the 10x more that aren't, however, this isn't a terribly large project.

I'm not knocking them for having a security flaw, and I hope I've not conveyed that I am. Everyone makes mistakes, bugs are a fact of software. What I am knocking is the fact that they handled the situation in a very non-user friendly fashion.

They could've, in fact should've, had a patch out in a far shorter time span. They've done so in the past, why not this time?

The answer is they decided to do a full code audit which took a week, and they found 8 or so more bugs, some of which might've been exploitable.

Isn't their motto effectively equivalent to "always auditing"? If so, why did this code audit find this much stuff? Why did they decide that a security hole was so bad that Theo needed to declaim the falling of the great blue dome, yet the fix could wait a full week?

Too many contradictions for me.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

Theo != user friendly (5.00 / 1) (#24)
by lb008d on Wed Jun 26, 2002 at 03:06:18 PM EST

Obviously the 2500 of which are running as root under privsep are easier to secure than the 10x more that aren't, however, this isn't a terribly large project.

You underestimate the practice of secure programming. Not only are you responsible for the 2500 lines of code being secure, but for each and every call those lines may make. That's why reducing the amount of privliged code is so important.

the fact that they handled the situation in a very non-user friendly fashion

Theo is famous for this. He's out to write good, secure software, not to please anyone.

If so, why did this code audit find this much stuff?

When you're intensely looking for one thing, it's often the case the other bugs come to light much more quickly.

Look at it this way - Theo says "Just disable Challenge/Response, and use our quick fix to patch the integer problem". This alerts all of the people out trying to make exploits to the critical section of code being worked on, and one of them creates an exploit from the other bugs that weren't fixed (but would have been within the week). Now Theo has to announce another security problem when those other 8 or so bugs are found, and the cycle is repeated.

Even if Theo says the sky is falling, a responsible admin will weigh all of his options before rushing into a solution. Nobody is forcing anyone to upgrade here. Here's how I think a smart sysadmin would have handled the situation.

[ Parent ]

ISS had a laughable response (5.00 / 4) (#12)
by El Volio on Wed Jun 26, 2002 at 02:09:43 PM EST

Essentially, since Apache is open source, they decided that it would be a security problem to notify them before they notified the world. So what they've said is, "if you use open source software, we're going to do our best to screw you over by telling everybody about the problem before the developers can fix it". Their defense is that they released a patch; unfortunately, this didn't consider operational issues (like the fact that their initial advisory focused on Apache for Windows but almost no one compiles that themselves for Windows) or thoroughness (as mentioned in the article, the initial patch fixed only a very small part of the vulnerability).

ISS did not follow responsible vulnerability disclosure practices in the Apache case. Suggestions have been made that it was because of past grudges against Red Hat (who employs some of the Apache developers). But they say it was because of the openness of Apache development. Wonder why they chose to handle OpenSSH differently?

Theo's behaviour is always questionable... (3.66 / 3) (#19)
by mingofmongo on Wed Jun 26, 2002 at 02:50:19 PM EST

it is seldom answerable.

"What they don't seem to get is that the key to living the good life is to avoid that brass ring like the fucking plague."
--The Onion

Just to clear the air on why this went this way (5.00 / 2) (#27)
by danimal on Wed Jun 26, 2002 at 03:35:45 PM EST

From the OpenBSD misc list and straight from Theo:
> What might have been nice(!) would be a 'use privsep, or > disable challengeauth, whichever suits your problem-space > best'. Privsep didn't even 'fix' the problem, just reduces > its impact. Turning off the challengeauth merde does (if I > understand the ISS alert anyway). Saying "disable ChallengeResponse" would have focused attention immediately on about 500 lines of code.

I don't know about any of you folks, but having that out there before the fix is available is just too frightening to think about. Theo did the right thing and gave people and vendors a chance to get a block up before script kiddiots got ahold of something. Do you not realize that this is what happens with BIND and the like?
--
<bestest> what does the dark side lead to
<@justinfinity> a gleeful life of torturing people and getting your way

doh, screwd my quote (none / 0) (#28)
by danimal on Wed Jun 26, 2002 at 03:37:33 PM EST

> What might have been nice(!) would be a 'use privsep, or
> disable challengeauth, whichever suits your problem-space
> best'. Privsep didn't even 'fix' the problem, just reduces
> its impact. Turning off the challengeauth merde does (if I
> understand the ISS alert anyway).

Saying "disable ChallengeResponse" would have focused
attention immediately on about 500 lines of code.

--
<bestest> what does the dark side lead to
<@justinfinity> a gleeful life of torturing people and getting your way
[ Parent ]

Bah (4.00 / 1) (#35)
by Miniluv on Wed Jun 26, 2002 at 04:25:38 PM EST

That's a pretty lame defense. If that fixed the problem then why is it an issue if the exploit is out there? When my software vendors recommend workarounds and say patches are forth coming, I decide if I'm exposed, and if so I follow their workaround. Anything else is stupid and unprofessional.

Plenty of situations like this have cropped up in the past, and when the fix was going to be difficult, but the risk of exploit was high, vendors have struck this middle ground that protects their users while buying them time to patch.

And Theo's on crack if he doesn't think that his announcement didn't narrow down the realm of the exploit significantly.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

Re: Bah (3.00 / 1) (#39)
by danimal on Wed Jun 26, 2002 at 05:04:30 PM EST

What makes you think they knew that setting ChallengeAuth to no would disable the exploit at first. Maybe it took them until today when they committed the code. They were able to tell that PrivSep defeated it tough.

You sure are doing a lot of hot air generating here mister, but until you come out with (a) a better product or (b) a track record of security then I don't know what you're screaming about. If the section of the code was so easy to figure out from the original announcement then where is your patch for the bug? Going from ~2500 lines to ~500 is a big deal.

I think trhurler's right, you just have a problem with Theo. Please get over it.


--
<bestest> what does the dark side lead to
<@justinfinity> a gleeful life of torturing people and getting your way
[ Parent ]

No (5.00 / 1) (#41)
by Miniluv on Wed Jun 26, 2002 at 05:20:15 PM EST

First, I didn't say I knew which ~500 lines of code it affected, but it was pretty obvious that it had to be a preauthentication overflow. Privsep is designed primarily to protect against exactly that scenario.

Second, if the overflow is when doing Challenge Response authentication, 5 minutes of testing would've indicated if it was overflowable set to off.

Finally, my objections to Theo as a person are irrelevant, as what I'm focusing on was his handling of this particular incident. He posted an inflammatory, misleading piece of information to bugtraq and I find that unacceptable.

"Too much wasabi and you'll be crying like you did at the last ten minutes of The Terminator" - Alton Brown
[ Parent ]

ah, opinions (none / 0) (#44)
by danimal on Wed Jun 26, 2002 at 05:24:12 PM EST

like assholes they all stink.

I think you are wrong and that Theo did a good job here, but you are more than entitled to your opinion.

enjoy it (but do calm down a bit).
--
<bestest> what does the dark side lead to
<@justinfinity> a gleeful life of torturing people and getting your way
[ Parent ]

Did they know? (4.00 / 1) (#50)
by ocelot on Wed Jun 26, 2002 at 08:45:12 PM EST

Theo said "Saying "disable ChallengeResponse" would have focused attention immediately on about 500 lines of code."

While not proof that they knew about the solution to begin with, it does kind of imply that they knew about it, but chose not to disclose it for fear of leading people to the problem.

Another solution might have been to say "Disable all unneeded methods of authentication". While still narrowing the focus, it's a good deal less specific.

But 20/20 hindsight, you know? :)

[ Parent ]

sort of (4.00 / 1) (#49)
by ocelot on Wed Jun 26, 2002 at 08:22:23 PM EST

In many cases, I'd agree with you. In this particular one, I don't.

Upgrading to 3.3 and enabling privsep is really not a valid solution. It worked on one of the three boxes I upgraded, and required a non-trivial time investment, both in terms of the actual upgrade and troubleshooting the non-working boxes.

Disabling Challenge Authentication took 10 minutes, most of that due to the oddness of one server which required trips to the server room.

While I agree that providing the Challenge Auth solution right off the bat would have been dangerous, I'm forced to question Theo's insistance that everyone update to 3.3 immediatly ("However, everyone should update to OpenSSH 3.3 immediately, and enable priv seperation in their ssh daemons"). He implied strongly that this was the only option, when it was not, and I suspect a lot of unnecessary time and effort was put into reacting to this alert.

This is especially true considering that 3.3 was an acknowledged interim solution, and that the problem would not be fixed in 3.3. However, he made no mention that 3.4 was in the works, let alone close enough that it would be released within a few days.

His original email seems more like an attempt to bitch at vendors than an attempt to actually help those of us who have to work with the thing.

[ Parent ]

It goes against proper release management. (5.00 / 1) (#51)
by Keepiru on Wed Jun 26, 2002 at 11:01:13 PM EST

I'm a network administrator for a large company.  I am responsable for a large server infrastructure which is based on Debian GNU/Linux, deployed domesticly and internationally wherever our network goes.  Flaky software isn't just an annoyance for me.  If I deploy a bad patch and lock myself out of even a small percentage of my servers, I'm royally screwed.  In many cases, I don't even have IT staff within a thousand miles of some of our small field offices to log in on the console to fix it.  That's why I run Debian in the first place - they have an incredible record of software quality.

So, when Theo comes along and declares that I have a security hole, and that the way for me to fix it is to upgrade to the newest version of his code that he released a couple days ago, and to enable the new feature that he's just finished working on, just to limit the damage to compromising a chroot-jail account instead of root, I just have to laugh.  Beta-testing his latest pet project is not an acceptable risk; it's a joke.

What *should* he have done?

Well, in his message to bugtraq, I think he spent more time talking about how terrible it was that all the distro vendors weren't falling all over themselves to play along with his joke, instead of giving us the key bits of information we needed.  Specifically:

What versions are affected?  (The version that ships with the stable version of Debian isn't even affected by this.)

The fact that there would be a workaround available to prevent the problem without upgrading.  This would have been very valuable information for determining the need to upgrade.  He wouldn't even need to specify which features, as long as I knew that there would be a workaround.

Instead of just telling vendors that the sky was falling and that the only way to save yourself was to upgrade to hot-off-the-skillet code, he should have provided them with the patch (which is a very simple one involving only a few lines of code) so they could incorporate it into the versions of SSH they ship.  QAing a backported security fix is a lot easier than QAing a huge jump forward in software versions.  This is how Debian usually handles security - they backport the fix into the production code.  It keeps me secure without having to play the version-of-the-week game.

Any of these would have given everyone enough to work on to evaluate how they wanted to proceed.  None of these options would have required disclosing exactly where the problem was to the public, or even vaguely where in the code to find  it.

IMHO, it looks to me that the real reason he's handling it this way is that he wants to coerce us into upgrading all our old software to his latest and greatest, for whatever reason.

This goes completely against the controlled release cycle that is necessary to maintain high quality software.  I believe this is one of the reasons he's getting flamed so hard for this release.

--Kai
--slashsuckATvegaDOTfurDOTcom

[ Parent ]

I don't know how you got two 5's (none / 0) (#56)
by tzanger on Thu Jun 27, 2002 at 11:08:53 AM EST

I don't know about any of you folks, but having that out there before the fix is available is just too frightening to think about. Theo did the right thing and gave people and vendors a chance to get a block up before script kiddiots got ahold of something. Do you not realize that this is what happens with BIND and the like?

The (stopgap) fix (disable challenge auth) is simple to do. Sure, 500 lines of code is simple to go through, but developing the exploit would take a day or two. You're telling me you can't add a line to your vulnerable servers in that timeframe? Hell I've got about two dozen I had to do that to and it took about an hour or so with a few phone call interruptions.

If I read the announcement correctly it's an integer overflow. I've been a C programmer for well over 10 years now and I think that it would be very very difficult, if not next to impossible to gain remote root through an integer overflow. We're not talking about smashing the stack here, we're talking about getting a number less than you expected. An immediate exploit that comes to mind is to somehow wrangle that number into a UID variable or code which checks for a return status. Doable but very very unlikely.



[ Parent ]
Not just ChallengeResponse (none / 0) (#57)
by uhoreg on Thu Jun 27, 2002 at 05:14:55 PM EST

As it turns out, there was some PAM code which was buggy as well (which ISS failed to mention in their advisory), so just saying "disable ChallengeResponse" wouldn't have completely solved the problem. I think that's why Theo made the first announcement — he wasn't sure that that was the only bug, but he know that privsep would prevent whatever remaining bugs were out there from being exploited. Of course, he did embellish his announcement, but overall, I think he did the right thing.

[ Parent ]
Many screwups in these situations. (4.00 / 1) (#40)
by xrayspx on Wed Jun 26, 2002 at 05:19:20 PM EST

The Apache project underestimated the scope of their vulnerability. True.

Theo went very deep-end and made this seem much worse than it potentially was. Also True.

In both cases we have ISS releasing exploit information WAY too early for a proper vendor response.

I do agree with Theo's actions, even after it came out that it was easy enough to work-around the issue with current OpenSSH versions. He had the correct instinct, which is prepare for the worst, always.

Apache hadn't really had time to deal with their vuln either before X-Force dropped their advisory. I don't believe they have followed the accepted guidelines, as loose as they are.

Maybe it's that Microsoft has more experience in this area than the others, which may be why surprisingly few people complain about their slipshod style of releasing a patch well after an advisory, which, by the way, breaks other functionality and, oh yeah, may not fix the original problem.

So yeah, no one acted perfectly, under the circumstances though, I think that OpenSSH team, and XForce are the least accountable. ISS is under no obligation to keep anything quiet for X length of time, except as professional courtesy, and OpenSSH playing up a vulnerability gets people to install the patch when it comes.


"I see one maggot, it all gets thrown away" -- My Wife
its BSD, have your way. (none / 0) (#53)
by elias on Thu Jun 27, 2002 at 07:21:04 AM EST

If the latter then stop releasing the portable version and let the community fork it into something that actually makes a solid attempt at cross platform support. Then we can rip out flaky and dubious features like privilege separation and any other similar schemes that might be dreamed up.

It's released under a BSD license. Feel free to do as you say.
Somehow i don't think you will.

I mostly liked the way Theo dealed with the proble (4.00 / 1) (#59)
by ersatz on Fri Jun 28, 2002 at 08:31:12 AM EST

On the whole, I think Theo dealed with the OpenSSH vuleribility rather well. One thing to keep in mind here is this is Theo De Raadt we're talking about here, the guy who leads the development of one of the most paranoid operating systems. It should probably be assumed he's going to choose the most security-paranoid path possible. Not disclosing the exact nature of the vuleribility initially was definately a good idea, as it gave people time to secure their machines. A lot of people would have probably been rooted if he had done otherwise. As for not telling people that they could secure their machines by turning ChallengeResponseAuthentication off, and rather telling people to enable privilige seperation, I think he did the right thing, even if it was really a pain in the ass. Doing otherwise would give away too much information about the nature of the vulneribility. And I don't really blame him for getting pissed off at vendors for not getting privilige seperation code working. It's really a good idea. The only thing that particularly rubbed me the wrong way was the way OpenSSH 3.4 was released as the fix. On a production machine, running an untested package like that would most likely be a bad idea. I would have prefered a backport of the fix to 3.3 for those who actually need ChallengeResponseAuthentication on. Personally, I just turned ChallengeResponseAuthentication off, as that, to my knowledge, completely negates the security hole. That, and it's not exactly a production system I run :). I don't know about the Apache thing, though, so I can't really comment. P.S. My bad, at the end of the disclosure it gives patches. Okay, so, I actually do think he did things properly.

Mistaken Disclosure | 59 comments (49 topical, 10 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!