Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
WebSource - a web based software development environment

By azul in Internet
Tue Feb 13, 2001 at 08:22:23 PM EST
Tags: Software (all tags)
Software

Here I introduce an interesting idea I've been thinking about in the last few days. This is a web-based software development environment, aiming to substitute your text editor, CVS, bug-tracking systems and other tools, fully integrating them. You could, in the same window, see a function's code, its assertions (separated from the code), good documentation about the way it works, links to other relevant parts of the program, a list of bugs linked to the function, a set of messages posted to the function, links to all the earlier versions of the function (along with information about their author and the dates on which they were written) and more. WebSource also borrows many ideas from literate programming and uses them exhaustively. Even though WebSource is not even vaporware, I'm posting this to Kuro5hin hoping to generate some discussion about its design that make help me in the future during implementation (in case this needs to be mentioned, if I implement it I'll release it under a free license, probably the GPL).


I have this idea that I decided I would write down and let others review for some time before I actually begin to implement it (if I do). I must state that I'm not sure this program is actually worth implementing, but I find the idea interesting.

Ladies and gentlemen, I give to you WebSource.

Okay, only its requirements.

WebSource is an innovating interface for developing software. In this paper I will review some of the features it should have once ready. I will also suggest some other interesting ideas that I have and ask for your feedback.

Why is it so innovating? Lets make a quick list of its features, that I'll explain with more details bellow.

Web-based interface

I know some of you might get a "YUCK that would suck" reaction. Just wait, let me explain all the benefits of a web-based interface. But yes, you edit the source code using HTML forms.

Versions Control / Central Repository

Okay, this on it self can't be called innovating. Its integration with the other features certainly is. No need for CVS if you use WebSource.

Bug Tracking System

Some of the advantages of the web interface: it allows us to perform bug tracking efficiently. It would be simple for programs to automatically report bugs to the central database (more on this soon).

Literate Programming

It incorporates most of the ideas common in tools for literate programming. Basically, it tries to add information to the code. This information is, however, not limited to documentation but includes many other things that we'll see soon. Also, different than most current literate programming approaches, it will not be designed to produce printed documentation but to produce it in an efficient hypertext system.

To be exact, WebSource is an application that runs on the web server (lets say via Common Gateway Interface; in reality it'll be something more efficient such as Apache Modules). It lets developers work on a given program, using their web browser as the interface used to edit its source code. It will maintain the code in the system in the web server.

This system is designed specially for developing free software. Though proprietary software might benefit as well by its use, we are focusing on the typical development of large free programs.

Using WebSource, we view a program as a group of many small interrelated programs. They don't have to be functions, just small groups of code that do something. We will refer to them as sections. If you don't understand the meaning of sections, refer to standard documentation for literate programming.

When developing software using most of the common literate programming tools, you end up writing a whole source file where it is impossible to see the logic structure of your programs efficiently. You have to actually produce the documentation (a process known as weaving) before being able to see things such as the relations between the different sections of your program. We feel that this information empowers the developer and should be available at all stages of the development.

Basically, your whole program is the big section, and its broken down into smaller sections. You have a tree where each section can have as many sub-sections as you desire. The leaves in the tree represent sections that have nothing but code written in your programming language of choice; all other nodes can contain both code and references to other sections (this references work as inclusions of other sections; think like function calls or like C-preprocessor macros that are defined as functions).

WebSource will be language independent. It will support as many programming languages as possible, each independently described in individual configuration files. Initially, we plan to add support for either C, Ruby or Java, eventually adding other programming languages as they are demanded by the community.

Lets now look at the information kept for every section of code kept in control by WebSource:

Version Control

As the program evolves, the different stages of every section will be kept. For every different version of a section, we will keep the author's comments (stating the reasons for the change) and the date it came into effect. It should be possible to see all the different versions that have been available for every section.

Parameters

Unlike other literate programming tools (specially CWEB), we will make it possible to pass parameters to the sections. This will work pretty much like the parameters to C-preprocessor macros defined as functions: all the occurrences of the name of a parameter will be replaced with whatever the caller section passes. This is related to the assertions, defined below.

Source Code

Here goes the actual source code for the section, along with a mechanism for inclusion of other sections inside the current section's code.

Special comments by the author

The author should state what the code does and why she choose to write it the way she did, along with any useful information.

Forum for discussion

We feel that by having a forum where everyone can post notes, we will encourage people to discuss about the different features and bugs in every section. This should make it easy, for example, to suggest different alternative algorithms. The forum should be a little smarter that most common message boards: for example, it should make it possible to mark a given message as a bug report (with different priorities for each bug report). Most forums would be empty, but others might generate a lot of discussion.

Links to other sections

When viewing a section, it should be possible to see other sections that (a) the current section depends on as well as those that (b) depend on the current. This should make it very easy to navigate the source code.

Assertions (preconditions and postconditions for every section)

We will provide a special field for preconditions and another for postconditions. This takes the form of source code. They could be included in the "Source Code", but we feel it's appropriate to separate them so depending on the type of build the program can be compiled with or without them.

Keeping them separate from the actual source code might make it easier to find Heisenbugs: side effects of the assertion checks that cause bugs when they are not included.

Finally, a third advantage of keeping assertions separated is that this facilitates programming by contract.

The system for the assertions must be such that whenever an assertion fails, the whole ``stack'' is reported (here, the ``stack'' is defined by the sections, not actual function calls).

The following features are highly desirable in WebSource. Some would be easier to implement than others. Some of them we are not sure deserve implementation, but we still find them interesting enough for inclusion.

Syntax Highlight

It would be desirable for WebSource to be able to display the code performing syntax highlighting. As the actual highlighting of a section depends on the part of the program where it was found and this information might not always be available for a given section, this might turn out hard to do reliably: WebSource would have to check the places where a given section is used to perform the highlight. Should this feature be implemented, we feel that an already existing mechanism for performing syntax highlighting is reused. Vim's syntax description files come to mind.

Error Detection

WebSource should automatically make checks on the code submitted to see whether it causes errors. This information should be related to the versions' control.

Automatic Indenting

For programming languages that don't care about the way a program is indented (almost all, but Python, for example, depends on the way the code is indented), every programmer should be able to state her preferred indentation style. WebSource would automatically adjust the code for her preferences.

Automatic Building capabilities

WebSource must be able to automatically build an executable from the source at any given time (assuming, of course, that the code does compile).

It must be easy to download either (a) the resulting source code when all the sections are integrated (this process is usually known as tangling) or (b) a binary program (ideally for whatever platform the user chooses).

It must be possible to download this results compressed using different algorithms.

Unit Testing

WebSource should allow developers to add different checks on the different functionality provided by the program. We borrow this idea from Extreme Programming because we think it works very well. Whenever a new version for a section is submitted, all the unit tests will be run and the results added to the versions information.

For very large programs, it will be possible to disable this feature and have the checks be performed periodically (every night at 4 AM, or something).

It must be easy to know the results of all the tests at any given time.

Object Oriented Programming (OOP)

This is a very popular programming methodology that we want to support as well as we can. We believe that the system as is defined blends very well with OOP, but we will listen to any suggestions that might improve it.

Use of XML to provide interfaces to other programs

It should be possible to export all the information in XML to make it easier for agents other than a browser to interface with WebSource. That way, anyone could design agents other than the web browsers and access all of WebSource's functionality through them.

Use of HTML to display and edit code

Some might see this not as a feature but as a necessary problem of our system. However, this might actually make a very nice editor. Consider, first, that most of the sections will never be bigger than a few lines. Also, consider that we can make hypertext links when displaying code from things such as variable names, types, functions and so on. Just think about all the links that the following C code could have:

<pre>{ MyNode *i; for (i = arg_head; i != NULL; i ++) if (!mytype_equal(i->val, current)) mytype_display(i->val); }</pre>

We can think of around 6 different links. And no, the code wouldn't look all in blue and underlined, we would use good HTML and CSS to ensure that the links are not intrusive and don't make the code hard to read.

Batch Edition

Okay, so you are just starting and want to create 200 sections with code, and you'd rather do it with your favorite text editor than with poor browsers' textareas? Sure, it should be possible to do that and easily integrate the resulting text file with WebSource (say use the browsers Upload button to update the program: WebSource then checks the file uploaded and, if it doesn't have any errors, it integrates the changes).

To make this easier, it should be possible to export-out sections: you should be able to check any given number of sections and your browser would download a file with them all that you can edit using your favorite text editor to update them all at once.

Locking Of Sections

This is required to allow many persons to work on a given program at the same time.

Locking should be used at many parts of the WebSources system.

Bug Reporting Mechanism

WebSource must come with simple interfaces that allow users with no programming knowledge to report bugs. The bugs can be eventually linked to sections by the developers. The standard information for every bug report is kept: platforms affected, priority, status, reporter, date when it was reported (and all the dates when its information changes) and so on.

Another idea here, specially related with the assertions information for every section, is to have WebSource build programs in a way such that whenever an assertion fails, the program automatically reports it to the server using HTTP. As the actual implementation for this depends on the programming language as well as the kind of program, this must be implemented for every program (though we believe code reuse will certainly become a pattern here) as a section that gets executed whenever an assertion fails.

Stats about the Development

It should be possible to export statistics about the development of the program such as the total number of lines written by each programmer or the average number of lines per unit of time (here the time would be the time during which a section is locked).

Graphical Information on the Program

WebSource should be able to render graphs with useful information about the program. This includes the following:

  • Graph about sections usage, so one can know which are the most used sections. And which sections are not used anywhere. Stuff like that.
  • Tree view for all sections. It should be possible to view all the sections as a tree. The user should be able to specify the desired depth of the tree.
  • Results of Unit Tests over time. This could be used by some as a way to measure the progress of the project.
  • Number of bug reports as a function of time. This would make a lot of sense if the automatic code for reporting assertion fails is used.
  • Language dependent graphics. For example, for OOP, it should be possible to see graphs for the different classes (such as the sections used by each method of a given class).

Feel free to suggest others.

Email Interface

This would allow the users to receive information on the status of the program periodically at their email. For example, reports on the bugs or the changes to the sections. It should be possible for to reply to those mails and have WebSource automatically incorporate the content or forward it to the appropriate persons.

User Authentication

WebSource needs a good system of accounts giving each user different privileges. It would be interesting to make it so a given user can only edit certain sections. Also, this system should keep settings such as the preferred indentation-style for every user or their email preferences.

Mozilla and advanced features

I think technologies such as JavaScript are not worth using. However, other persons might disagree and argue that some special features of Mozilla, not available in other older browsers, should be used. I think WebSource should be designed so none of its significant features is missing in a text-only non-JavaScript browser like W3M.

Using WebSource would not be a win-win situation, but more like a sort of tradeoff. Specifically, writing new code might become slower as you are forced to add information to it. However, it might make it easier to anyone interested in modifying the software to understand it, find the relevant code to implement his modifications and find out if his changes break any other sections of the program. This should, in the long term, make software more maintainable.

Anyway, that's pretty much all the ideas I have for now, and it should give you a clear view of what WebSource will be (assuming it eventually is). I welcome any comments and criticism at my usual email address, bachue@bachue.com.

I will not work on WebSource for some time, hopefully giving you enough time to contact me and send me all your suggestions and criticism.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Kuro5hin
o Vim
o Also by azul


Display: Sort:
WebSource - a web based software development environment | 17 comments (12 topical, 5 editorial, 0 hidden)
Hmmm (3.00 / 3) (#5)
by jabber on Tue Feb 13, 2001 at 05:12:14 PM EST

It sounds a whole lot like Keystone with an integrated IDE. I personally don't care for too much integration, especially in distributed (web based) applications, but this is a really innovative idea you've got there.

How about folding in a tool for UML diagraming, and maybe round-trip engineering between diagram and source?

Also: This is the sort of thing I want to see on Kuro5hin! This, Carnage's excellent story and breaking news - with a good critical write up. No MLP. No Hellmouth derivatives. No Fawking Trolls. And no political/philosohical mumbo-jumbo. Those belong on Memepool, slashdot, geekizoid and plastic respectively.

[TINK5C] |"Is K5 my kapusta intellectual teddy bear?"| "Yes"

Modules. (3.00 / 2) (#7)
by Seumas on Tue Feb 13, 2001 at 05:17:15 PM EST

I think the important thing would be not to address all these issues in one massive piece of software, but to break it up into very adaptive modules. Use what you need. Substitute better solutions for one piece or another (assuming the substitutions were also easily adapted). The One Big Solution, rarely ever is. But lots of little solutions, with The One Big Solution in mind can be pretty powerful.
--
I just read K5 for the articles.
[ Parent ]
Ah, no... (4.33 / 3) (#8)
by trhurler on Tue Feb 13, 2001 at 05:44:38 PM EST

No MLP. No Hellmouth derivatives. No Fawking Trolls. And no political/philosohical mumbo-jumbo. Those belong on Memepool, slashdot, geekizoid and plastic respectively.
Nobody but trolls would argue that trolls should go away. Hellmouth crap is no longer amusing. However, much "political/philosophical mumbo-jumbo" does in fact form a part of the technical culture, and that's topical here. And, of course, if MLP were not appropriate, there wouldn't be a damned section for it. Don't like it? Don't read that section. Very simple.

There's a mission statement, there's a FAQ, there's lots of posting history, and yet people still seem to want to redefine k5 to be some kind of all-tech-all-the-time fantasy. Why? This makes no sense.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
mumbo-jumbo (4.00 / 1) (#10)
by Seumas on Tue Feb 13, 2001 at 05:58:47 PM EST

I won't presume to speak for him, but I think what he may be eluding to as 'mumbo-jumbo' is the political submissions which amount to little more than a transcription of a property-line action pamphlet, which in turn calls for little more response than line-towing mumbo-jumbo and opposing mumbo-jumbo. We see it all the time -- submissions, posts and replies that are really not founded on the true viewpoints of the posters (if they have viewpoints), but on what their favorite talk-show host or political proponent wrote/said/initiated somewhere.

One should note that the slogan is "technology and culture, from the trenches" and not "technology and culture, passed-on from the propaganda machine / television news / whoever else gave me an opinion". There are some good political posts on here, but they are few and far between.

But hey, that's just me. I dunno. I am easily annoyed by the same old arguements to the same old questions, but easily entertained by new viewpoints to the same old arguements. Re-hashing the same "i'm right, you're wrong" debates usually doesn't lead anywhere and isn't enjoyable or education for anyone involved, unless honing your subtle torching abilities.
--
I just read K5 for the articles.
[ Parent ]

Clearer reiteration (3.00 / 1) (#11)
by jabber on Tue Feb 13, 2001 at 06:17:02 PM EST

This is the sort of thing I want to see on Kuro5hin! [...] No MLP. [...].

Now that that's cleared up..

What I mean by "mumbo-jumbo" is the constant banging of the same heads together. The same BS dressed in slightly different words. And the arguing of semantics based on how essoteric and obscure someone's reading list was. When a story demands that every word be preceeded by a qualifying definition, the whole point of community conversation is lost, and the discussion becomes a dissertation or a legal brief.

[TINK5C] |"Is K5 my kapusta intellectual teddy bear?"| "Yes"
[ Parent ]

Interesting thought...consider also: (3.50 / 2) (#9)
by Erbo on Tue Feb 13, 2001 at 05:45:55 PM EST

If you could make it so that WebSource runs even if you just have a Web server running on your local machine (i.e. you're pulling the Web pages from "localhost"), then you would have an even more powerful tool, which could let you move a project from local (one-person) development to collaborative development in a seamless fashion, or let people work on "snapshots" of a collaborative project locally and then "synchronize" the project's master data later.

It's not as crazy an idea as you might think. Many operating systems include at least one elementary Web server that may not be "industrial strength," but is certainly good enough to use by one person locally. (Some include more powerful ones, too, of course.)

Another option might be to include some sort of instant messaging capability, such as Jabber, to allow multiple developers to get up-to-the-minute notifications of new checkins or forum messages.

This is an interesting project idea, and I encourage you to go forward with its development...I think it's something unique.

Eric
--
Electric Minds - virtual community since 1996. http://www.electricminds.org

Just leave me hanging now (4.00 / 2) (#12)
by bjrubble on Tue Feb 13, 2001 at 06:18:11 PM EST

Just wait, let me explain all the benefits of a web-based interface.

I looked through the rest of the article, and didn't see this promised explanation.

Okay, there's the linked display of code. That's actually pretty cool; I love the Qt docs in part because they're very hyperlinked, and if applied to the code itself this might be even more powerful.

But you still haven't given me any reason to consider that editing in an html box is going to be any less a complete pain in the ass than it has been every other time I've had to do it.

I believe Zope has the right idea here. You can get an emacs module (and probably something for other editors) that allows you to check modules in and out via ftp/ssh, making the editor integrate relatively seamlessly into the rest of the development environment.

Even so, I'm still deeply skeptical of working on code that's not local. Unix has a wide range of extremely powerful tools that allow you to do pretty much arbitrary operations on large collections of text, and do them very quickly. Every other system I've worked with has been extremely painful to use in comparison.

And one last comment unrelated to the rest of my post:

WebSource must come with simple interfaces that allow users with no programming knowledge to report bugs.

You could do this and nothing else, and I would worship you. If Bugzilla really is the best bug system around, that's just sad.

Random thoughts: (3.66 / 3) (#13)
by cezarg on Tue Feb 13, 2001 at 06:20:11 PM EST

Cool: This idea has a lot of potential. If it's nicely integrated it might be a really nice approach to the platform independence issue.

Not cool: The biggest technical challenge would be to provide a decent editing widget. All the ones I've seen are not suitable for code editing (they lack even the basics like brace matching).

Cool: The real strength would be the collaborative aspect of the project. A really neat feature would be "remote compilation". I write my code on my peecee and hit "Build remote" and it goes off and builds the project on my friend's PowerPC. this way I could have more confidence that the stuff I'm doing isn't gonna break someone elses build.

Not cool: Latency. Bandwidth is getting cheaper and better but the latency of packet switching might slow ceratin things quite a bit. Since you'll be relying on the network side of things more than traditional IDEs this might be something to watch.

Cool: Real time collaboration. A neat feature would be to see who's logged on and coding at any instance and be able to discuss code with them.

Cool: All the stuff you mentioned in the article.

Overall I think you have a superb idea (I'm surprised noone has tried it yet). There are certainly challenges you'll have to face but the idea is there and I could see myself using it. Keep us posted

Flaws, but good (3.75 / 4) (#14)
by Aquarius on Tue Feb 13, 2001 at 06:21:25 PM EST

Excellent article, I must say.

However, I think that returning to web forms as a UI throws away the interactive nature of IDEs in this day and age. It's jolly nice to have as-you-type syntax highlighting. For those who like that sort of thing, it's nice to have Microsoft-a-like automatic command completion. It's nice to have split windows to copy and paste between them. It seems slightly nonsensical to me to throw away all the innovations that have been made in code editors and return to what amounts to being a pretty version of a 3270 terminal, just for back-end benefits.

Now, don't get me wrong. I don't disagree with the web as an interface. For starters, building web interfaces to applications and building applications that have web interfaces is what I do, both at home and for a living. I like the environment. But I don't think that it's right for editing, not basic web forms, anyway. I do not want to have to click a Submit button to see my code syntax-highlighted! We *can* have on-the-fly syntax checking of code if it's required. But not with the "make a change, submit to the server, wait for the result" standard web form method of processing, which really is like dumb terminals. Everyone moved away from that for a reason.

Some kind of server-side/client-side integration would be a possibility, but, as we all know, client-side stuff is notorious for being bad news, unportable between environments, and open to abuse by malicious clients.

All this aside, I love this idea to bits, although I'm a little puzzled as to why you intend to provide for C, Java and Ruby. Ruby? Why not Perl or Python?

How can I help with the project? :-)

Aq.

"The grand plan that is Aquarius proceeds apace" -- Ronin, Frank Miller
Well thought-out, but I don't think it would work. (4.75 / 4) (#15)
by GusherJizmac on Tue Feb 13, 2001 at 06:22:32 PM EST

Basically, hitting a web server is just too slow an awkward. The web interface would be just too annoying. The web doesn't easily allow real-time feedback of things you are doing. We have a web interface to CVS at my work. The HTMLers edit files over Network Neighborhood and the use this web interface to execute CVS commands on the server where the HTML is stored. The problem is that all they see is that their browser is churning, and they don't see ANY feedback about what's going on. Compare that to the command line where you can see info flashing by and see what's going on.

Also, I could kill myself if I had to edit source code in a text box or a Java applet (which are your only viable alternatives). I'm sure most other people here would as well. It's just too awkward.
<sig> G u s h e r J i z m a c </sig>
local devel platform (my first story comment =) ) (4.50 / 4) (#16)
by nickp on Wed Feb 14, 2001 at 12:00:12 AM EST

Think of Unix development practices: variety of tools for variety of tasks, pipes, invoke existing commands. You have a lot more flexibility when you can choose your tools and have finer granularity of the tools. I would recommend you to focus on the collaboration benefits and think about how to build a collaboration module that lets you use existing tools in a group environment.

The second issue is that web interface limits the flexibility of the developer. When IDE is running locally, it is intimately tied to the system on which I can test the program. This is absolutely necessary for any reasonable development cycle. If I can't test the code on the kind of system setup that my users will be using, I'm asking for trouble. If my testing is not tied to IDE it becomes inefficient to trace sources of problems (alternating back and forth between local system and stuff over the web).

You could say run the web server locally, but then it is still an extra level of indirection: not everything that can be done in IDE can be done efficiently in a browser.

I suppose these things can be fixed if WebSource could just automatically mirror the development done locally, upload it, convert it, etc.

I don't get the following:
When developing software using most of the common literate programming tools, you end up writing a whole source file where it is impossible to see the logic structure of your programs efficiently. You have to actually produce the documentation (a process known as weaving) before being able to see things such as the relations between the different sections of your program

Why is writing source where you can't see the logic structure good?! Documentation is crucial, but if the relationships of the code don't make sense without the documentation, this tells me there are serious problems with the code. I am hoping I am grossly misunderstanding what you meant to say.


"Gravitation cannot be held responsible for people falling in love." -- Albert Einstein

Not just your father's WWW. (4.00 / 1) (#17)
by slaytanic killer on Wed Feb 14, 2001 at 07:02:33 AM EST

You know, when I think "web-based," it's also server-based. HTML forms & Java applets are not the only views of a webserver. One can also use, say, a Java application (not an applet, but a real executable) which communicates with the webserver, and keeps various things on the local HD. Using the server when appropriate.

WebSource - a web based software development environment | 17 comments (12 topical, 5 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!