Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Need help choosing an embedded language

By ptomblin in Technology
Wed Jan 17, 2001 at 05:51:08 AM EST
Tags: Help! (Ask Kuro5hin) (all tags)
Help! (Ask Kuro5hin)

At my job, we're attempting to embed an interpreter into a multi-threaded server that runs on high end Sun hardware. I can't give out too many details about the project, but here are the requirements for the interpreter we're embedding. We've been trying to use Python, but are hitting a lot of snags and I was hoping that somebody might be able to suggest a language that better meets our requirements.


The language must be:
Embeddable.
We must be able to run the code from within a thread in a C++ program.
Thread Safe.
We must be able to have either interpreters running in several C++ threads without them interfering with each other, or separate threads of the interpreter running different blocks of code. And if one blocks, they must not all block.
Non Ugly, Easy to Learn, C or BASIC like
Note: My boss says that Perl, Lisp and Scheme are just too damn ugly (yes, they have their place, and I use Perl all the time. But this isn't the place for it). Something with fairly straight forward declarative language like C or BASIC would be best. Object oriented features are fine so long as they can be ignored by beginners. The target audience is probably most familiar with Visual Basic or PL/SQL.
Extensible
It must be able to call other modules written in the language, and it MUST be able to call functions written in C/C++. That C/C++ code may block waiting on an asynchonous process that may take a while, so the other threads other than the one calling that C/C++ code must continue to run.
Fast
We anticipate having several dozen of these threads running at once, and each one should be able to complete a script consisting of a few hundred or few thousand lines of code in a second and be ready to run the same script or a different one immediately afterwards.
Non-Graphic
The code will be running on a server deep in the bowels of a server farm with no graphics head on it. The language must be able to operate without graphics, and does not need any GUI or drawing constructs in the syntax of the language.

It would be extremely desirable if it would take a C string containing the block of code to interpret, rather than expecting it in a file. And if the block of code attempts to load another block of code, it would be extremely desirable if it would come back to our C code to get that block of code as a C string rather than expecting it to be in a file somewhere. We anticipate loading the interpreted code from another program as a series of IPC-style messages rather than a file. If the code can be pre-compiled or pre-parsed and passed into the interpreter as a byte-code stream or structure or array, so much the better.

It would be desirable if we could remove functionality from the language, both for simplicity and for security. The script writer is supposed to get his inputs using our own C++ coded language extensions, and do actions using our own C++ coded language extensions, and not do any file I/O or socket communications except possibly within a carefully controlled sandbox.

So what do you think? Is there a language out there that meets my requirements? I'd prefer something open source, because I've already experienced what happens to programs that rely upon a third party closed source tool that isn't available any more.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Also by ptomblin


Display: Sort:
Need help choosing an embedded language | 37 comments (32 topical, 5 editorial, 0 hidden)
Delphi... (1.57 / 7) (#1)
by sugarman on Tue Jan 16, 2001 at 11:44:39 AM EST

I think...

It seems that Delphi could fulfill most of your requirements (OO, can call C++ functions, work without graphics, etc), however I have *no* idea of it's suitability for an embedded language. It seems to rely heavily on the Windows API, or the Borland VFC, and I'm not sure how well that would scale down.

Other than that, PHP?

--sugarman--

You have NO idea what you are talking about (4.50 / 4) (#15)
by Dacta on Tue Jan 16, 2001 at 06:47:54 PM EST

(I used to be a Delphi programmer. Judge these statements accordingly)

Firstly, Delphi only runs on Windows. You may be able to get it running on Linux using Wine, but you have no hope on Solaris, especially on Sparc. Kylix (Delphi for Linux) will help somewhat, but that is still for Intel hardware only.

Secondly, Delphi is a compiled language, with a commercial compiler which you can't (legally) embed into other apps. That kind of kills the embedded language idea.

Thirdly, it's the Borland VCL, not the VCF, and it is quite possible to use the compiler without that. However, then you run into that pesky Win32 target problem, again.

Finally, even if you could get around the legal problems, the architecture problem and the it's-a-compiler-not-an-interprator problems, you'd run into the problem that the only real way to intergrate C++ and Delphi code is via DLLs (although I guess you'd reimplement that to shared libraries on Solaris) or COM/CORBA. I guess CORBA is a possibilty, but apart from that you are stuck with a funcational (not object based) interface between Delphi and C++.

I love Delphi, but this isn't the kind of thing it excells at.



[ Parent ]
Um (2.60 / 5) (#2)
by finial on Tue Jan 16, 2001 at 11:46:54 AM EST

I'd say PHP or APL. I like them both. It depends on whether you need very general purpose or whether you need some heavier math.

oo in php (none / 0) (#25)
by TigerBaer on Wed Jan 17, 2001 at 10:00:18 AM EST

The OO in php is not really OO. There is no variable hiding (i.e. private & protected ). Although this is not necessary for creating a single OO program, to create an OO library it is essential that the data members of an object are allowed to be hidden from other objects. This is one of the key concepts of OO, creating the modularization.

As to what embedded language to use, have you thought about basing your language on the best features of a few languages. Then use Lex && Yacc to create your interpreter.

[ Parent ]
PHPOO PHPOO (none / 0) (#29)
by finial on Wed Jan 17, 2001 at 12:24:10 PM EST

Yes, but he/she said:
Object oriented features are fine so long as they can be ignored by beginners.
So it didn't sound like it was needed. I have a soft spot for APL, though. Not the easiest to learn, but elegant and powerful.

[ Parent ]
Here's a nickel. Go buy a real language. (4.50 / 2) (#36)
by eLuddite on Sat Jan 20, 2001 at 02:55:12 AM EST

PHP is neither threaded nor thread safe. Its not very pretty, either, but I'm prepared to yield to others' opinion on that matter.

You are almost certainly looking for Pike which is the most superior, most underrated and easiest (for people with a C background) scripting language out there. Although you omit every possible detail of your X-File project, I doubt it will be as comprehensive as Roxen which is driven by Pike.

You are most welcome!

---
God hates human rights.
[ Parent ]

cint? (3.00 / 4) (#3)
by ana on Tue Jan 16, 2001 at 11:47:11 AM EST

You might want to try cint, which we've used for several embedded applications (like coding the function used in a fit-the-function-to-the-data engine, and for describing complex apertures and sources in a ray tracing code. Another choice might be lua which is C-like (but just different enough to be annoying).

Ana

Years go by; will I still be waiting
for somebody else to understand?
--Tori Amos

Well, Python it is then (4.80 / 10) (#4)
by spiralx on Tue Jan 16, 2001 at 11:57:40 AM EST

I'd say try and get past the snags, because it fits the all of your requirements pretty much perfectly apart from speed.

Python has always been designed with extending and embedding in mind, and it's pretty easy to do so. My first job involved a large management information system in which clients had embedded Python interpreters which ran scripts which in turn used wrappers around a set of CORBA objects representing various elements of a job. The initial time to implement the embedding was minimal and wrapping objects is generally more a matter of patience than skill or time... in fact I think there are automatic wrapper generators available.

Python also supports restricted execution through either the rexec or Bastion modules, and even without those you can create your own namespaces using dictionaries and have code execute in those rather than globally to ensure security. If you look at the code for MOOP, a MOO written in Python, it does a similar thing for user-defined functions using some cunning hacks...

Also, another thing MOOP does is compile Python code on the fly from a string and execute that, using Python's built-in compile() function, so you could pass in code in a C string.

Speed may be an issue because Python isn't particularly fast in itself, however if most of the leg-work is going to be done inside the wrapped C/C++ code then this shouldn't be so much of a problem. Just make sure any processor intensive stuff is done from the C/C++ stuff and the Python is used for controlling it.

And of course, Python is open source, so no problems there...

You're doomed, I'm doomed, we're all doomed for ice cream. - Bob Aboey

Good Idea (none / 0) (#17)
by Robert Uhl on Tue Jan 16, 2001 at 07:16:42 PM EST

I'll have to second that. I'm not too keen on Python myself--I'm a Perl man, and for some reason our camps have an inordinately small intersection set--but Python could be an excellent choice for your project. It really is quite a nice little language. Has some nifty features--'sposed to be very easy to learn. Might wish to take a look at it. Not my cup of tea, but it's differences that make horse races.

[ Parent ]
Wow, an unbiased Perl luser :) (none / 0) (#19)
by spiralx on Tue Jan 16, 2001 at 07:30:49 PM EST

Only joking... :)

But yeah, there does seem to be a lot of antagonism between the Perl and Python camps, despite the fact that the two languages do really have different roles to play if you look beyond the fact that they're both scripting languages. Perl is excellent for short scripts and text processing, both of which make it very useful for CGI programming, whereas Python shines more in larger projects and in the ease with which it interfaces with C/C++/Java (for JPython).

Of course, personally I think Perl is the work of the devil and looks like somebody has puked ASCII characters onto the screen, but we all have our prejudices... ;)

You're doomed, I'm doomed, we're all doomed for ice cream. - Bob Aboey
[ Parent ]

Multi threading (none / 0) (#32)
by jovlinger on Wed Jan 17, 2001 at 03:57:25 PM EST

I haven't looked for a couple of years, but last I saw, python wasn't the most multithreadable application around. Also, it will suffer from speed restrictions if a lot of the work is to be done by user written scripts.

I would suggest Lua, which isn't so much a language but a language construction kit. I've heard good experiences from a researcher who used it to develop a Software Engineering Code Repository.

YMMV of course, and I don't know how multi-threaded it is.

Johan

[ Parent ]
I hate tcl..... (2.75 / 4) (#5)
by daystar on Tue Jan 16, 2001 at 12:01:38 PM EST

But it is pretty nice for embedded systems.

Of course, that's messing with the Ugliness requirement. I don't quite get that one, though. I mean, I agree that perl can be ugly, but I also think that if your language choice is base on textual aesthetics, then you're probably doomed no matter what you choose....

--
There is no God, and I am his prophet.
Why not Java? (4.00 / 5) (#9)
by 0xdeadbeef on Tue Jan 16, 2001 at 01:17:54 PM EST

It has an interface between it and C/C++ (JNI).
It is certainly extendable.
It has a sandboxing API (java.security.*)
And will at least equal the performance of any embedded scripting language.
It's also easier to learn than Python, IMNSHO.

I could explore the feasibility of this further, but as others have said, I'd have to bill you. :-)

interpreted java (4.00 / 1) (#28)
by edric on Wed Jan 17, 2001 at 12:22:48 PM EST

There's a nifty little tool that i use all the time called beanshell which is basically a java interpreter. It's used by the JDE, a java development environment within emacs. JDE uses beanshell to look up info about java classes on the fly.

From the info page:

BeanShell is a small, free, embeddable, Java source interpreter with object scripting language features, written in Java. BeanShell executes standard Java statements and expressions, in addition to obvious scripting commands and syntax. BeanShell supports scripted objects as simple method closures like those in Perl and JavaScript(tm).


[ Parent ]
Rep. (3.33 / 3) (#11)
by ksandstr on Tue Jan 16, 2001 at 03:11:58 PM EST

If you can persuade the relevant suity types, I'd suggest rep. Even though it's a GPLed library (making it unusable if you're working from a binary distribution only-approach), it is really very good (take a look at sawfish the window manager, for example), particularly when used with the rep compiler.

Of course, LISP isn't sufficiently sexy anymore, so maybe you're just better off going with Python and dealing with the restrictions it has.



Fin.
Hopeless quest (4.50 / 4) (#12)
by tftp on Tue Jan 16, 2001 at 04:48:20 PM EST

I can't give out too many details about the project

And I can't have too many details to make a right choice! I doubt that you will get any useful advice this way, other than a random list of obscure languages. Designer chooses the language (as one of many choices) based on all information s/he can gather, and even then more is usually needed. It includes even such seemingly irrelevant details as capabilities of your product support team and software maintenance people. Only you can make the decision, and the research itself is essential; don't expect someone to offer a ready-to-use solution - none exists.

Ada? (3.75 / 4) (#13)
by fink on Tue Jan 16, 2001 at 05:32:23 PM EST

Real quick one, as I'm at work at the moment. :)

Okay, perhaps I haven't read your needs as fully as perhaps I should, but have you looked at Ada? Sure, it has some damn ugly features (but as a language designed by committee, you can expect that), but it's also quite a solid language. Most military contracts (in .us and .au at least) use it to some degree.

Ada is embeddable - although I am speaking of embeddable devices such as GPS and autopilot systems. It is thread aware and threadsafe. It is high level so (theoretically) very easy to learn. It is an OO language, so extensible. It is natively compiled, so fast. And it works well in non-UI (even without text!) environments. Yes, there is a GNU compiler for it.

I'd suggest that you use it at your own discretion however - "test" it first if you will. I've never found a use personally for it - I stick to Java and PHP myself for the work I do, and my experience with languages other than the "common" ones (C/C++/Java/PHP/Perl et al) is relatively limited.

My $0.02 ...


----

Re: Ada? (4.00 / 2) (#14)
by dreamfish on Tue Jan 16, 2001 at 06:14:30 PM EST

Sure, it has some damn ugly features (but as a language designed by committee, you can expect that)

A myth that frequently needs to be killed: Ada was not designed by a committee - the driving forces behind both versions were individuals: Jean Ichbiah (Ada83) and Tucker Taft (Ada95). Don't believe everything you read in the Jargon File :)

You failed to mention what you considered ugly? I hope it wasn't the usual "any syntax that doesn't look exactly like C++ is ugly". Bear in mind C++ isn't the ultimate implementation of OO (for one thing Ada handles polymorphism a lot better).

I can't remember the exact links but Ada has a good track record in being using in embedded systems. Have a look at AdaPower and The Ada Information Clearinghouse.

Finally, try it out with a gcc-based Ada compiler.

[ Parent ]

Ugly!?! (2.33 / 3) (#16)
by Robert Uhl on Tue Jan 16, 2001 at 07:12:28 PM EST

My boss says that Perl, Lisp and Scheme are just too damn ugly.

Much as I love Perl, I can take that criticism of it. But Lisp and its dialects (such as Scheme) are most emphatically not ugly. They are just about the most elegant programming languages around. Granted, they require a hyper-educated user. But that's the price one pays.

Seriously, I'd recommend either TCL or Scheme.

TCL would work well, I think (none / 0) (#18)
by Luke Francl on Tue Jan 16, 2001 at 07:29:59 PM EST

I prefer to program in Scheme myself (ugly?! Scheme? Whatever!), but if your boss is opposed to that, TCL will work just fine. It's thread-safe and pretty easy to learn (I just did ;).



[ Parent ]
Well, there's always Haskell... (3.00 / 1) (#20)
by tmoertel on Wed Jan 17, 2001 at 01:06:12 AM EST

You could always use the embedable Server API for the Haskell interpreter Hugs and use Green Card for your C bindings. Since Haskell is a purely functional language, many types of security-weakening errors are not possible in Haskell programs (such as buffer overflows, static buffer recycling, etc.). The sophisticated type system also catches many other common programming errors at compile time.

Plus, it worked well enough for mod_haskell.

It's sufficiently cool to merit a few minutes consideration.

On the other hand, if your boss thinks that things like lots of parentheses are sufficient grounds to dismiss a good embedable language like Scheme, you might have trouble selling him on Haskell's Monadic I/O. ;-)

--
My blog | LectroTest

[ Disagree? Reply. ]


The problem? (4.00 / 2) (#22)
by Ixokai on Wed Jan 17, 2001 at 06:01:14 AM EST

It would be very helpful if you told us what aspects of Python were causing the snags -- what the limitations are that you are hitting, what is wrong with the tool you are currently using so we can sugguest a better tool.

Global Interpreter Lock? (none / 0) (#23)
by ejm on Wed Jan 17, 2001 at 06:26:56 AM EST

Could it be that the global python interpreter lock is causing the problems? I suggest you try out Stackless Python, since that allows many lightweight threads via the uthread module, which is available via a link there (damn site is down :)).
It still doesn't break the global python interpreter lock (which by the way means that only one 'real' thread can interpret python code at a time).


[ Parent ]
Re: The problem? (none / 0) (#26)
by ptomblin on Wed Jan 17, 2001 at 10:31:08 AM EST

Well, as ejm suggested, our biggest problem is with the global interpreter lock. We just can't seem to find a way to either:
  • Run an instance of the interpreter in each C/C++ thread without them interfering with each other, or
  • Run multiple python threads where one calls a C function that blocks while allowing the other threads to keep going.
In the first instance we seem to have problems with global variables in the Python interpreter, and in the second one the one blocked thread blocks all of them.

--

My company is looking to hire 16 C/C++/Java Solaris developers in Rochester NY. Email (text) if interested.
[ Parent ]

Those are definitely solvable problems (5.00 / 1) (#31)
by alder on Wed Jan 17, 2001 at 02:44:34 PM EST

What you probably need is to find proper script execution paradigm, then you can do virtually anything (talking based on the experience of implementing something similar to what you described - though it's difficult to say how similar... :-).

The very first thing to consider is do you need to run threads? Though you can, you much better off with multiple processes then threads. In this case you do not have to manage (or have for that matter) global interpreter lock at all.

Secondly, if you really must use threads and they will call blocking C functions you must do some C programming around them. Just take a look at:

Py_BEGIN_ALLOW_THREADS
...Do some blocking I/O operation...
Py_END_ALLOW_THREADS
at 8.1 Thread State and the Global Interpreter Lock

The last: to avoid some (probably you cannot easily avoid all while using threads) problems with global variables try something like starting multiple subinterpreters, per thread for example (Py_NewInterpreter and Py_EndInterpreter at 8. Initialization, Finalization, and Threads). Subinterpreters run (almost) totally separate environment for the execution of Python code, so most of those global variables are not shared.

I think you've made a good choice selecting Python (well, I'm saying this based on as little knowledge about what you are doing as you provided :-). It's your execution paradigm which is probably at fault, not Python.

[ Parent ]

some pointers (4.33 / 3) (#24)
by joto on Wed Jan 17, 2001 at 09:28:22 AM EST

tcl, lua, Small, SIOD or librep.

Tcl could work, with some caveats... (4.00 / 2) (#27)
by Ricdude on Wed Jan 17, 2001 at 12:06:03 PM EST

We use Tcl at work for our embedded scripting language. We beat the snot out of it on a daily basis, and have found only one or two things we can't actually work around. Addressing your requirements individually:

1) Embeddable - You can indeed call Tcl scripts from C++ threads, we do this frequently.

2) Thread safe - Here is Tcl's (and i suspect any scripting language's) biggest snag. You can't perform parallel evaluations on the same "interpreter" from different threads. You can, however, mutex the interpreter when you need to perform evaluations from different threads. Or, you can create independent interpreters in each thread, and mutex the data access functions in your custom procedure calls.

3) Non-Ugly, Easy to Learn, C/Basic like - (Note: if your boss doesn't actually write code, they need to learn to work at the requirement/functionality level, and not micromanage the formatting of the source code) It's "like" C, in that it relies on curly braces for code blocks, but there are funny restrictions on whitespace usage that take some getting used to. If you want object-oriented features, there is a Tcl extension, [incr Tcl], for just that purpose.

4) Extensible - Tcl extensions are accessed via a "package" mechanism, e.g. "package require Apackage". You can access C and C++ functions (that have been declared extern "C")

5) Fast - Tcl now features a byte-compiler, and is capable of compiling scripts during the initial execution, and executing them from the byte-compiled form thereafter. As for a few hundred/few thousand lines of code in a second, your performance is going to depend on what's in that code as to execution speed, but as long as you're not doing anything too loop/compute intensive, Tcl should be fine.

6) Non-Graphic - Tcl itself is not a graphic programming language. The Tk extension provides a graphical programming interface for Tcl scripting. If your server is beried deep within a server farm, you could always use Tk to provide a GUI via X to a remote machine (we do this frequently at work).

Tcl resources:
http://sourceforge.net/projects/tclpro/
http://sourceforge.net/projects/incrtcl/
http://www.tcltk.com


Embedding (3.00 / 1) (#30)
by phuqwit on Wed Jan 17, 2001 at 12:58:43 PM EST

Have you considered the possibilty of using Inferno/Limbo as an embedded OS/language package? Vita Nuova supplies Inferno at a very cheap price if you want a physical copy and a manual, or you can download and compile it yourself.

Inferno is, as far as I am aware, gaining ground in the field of embedded programming, I think Lego uses it for some of their products. Additionally, it is VERY similar to C++, less of a learning curve for those who already know it, and it uses a great deal less code than many other programming languages (76 line gif view, last time I checked).
=== You may or may not need to reboot in order to use this feature of Windows.
Pike (none / 0) (#33)
by mwilson on Wed Jan 17, 2001 at 04:43:17 PM EST

What about Pike?

http://pike.roxen.com

It covers all your requirements, and it's GPLd. check it out. (sorry for the short port, gotta work)
Matthew

Tcl (none / 0) (#34)
by ritesh on Fri Jan 19, 2001 at 09:49:28 PM EST

was designed as an embeddedable language. For instance, Motorola used to embed a tcl interpreter in their cell switches. The useful thing with Tcl is that using a tool like SWIG (Simplified Wrapper Interface Generator) you can easily wrap C/C++ code into Tcl. Then you can write Tcl scripts that calls those C functions. I would recommend papers on the philosophy on Tcl by John Ousterhout; the author of Tcl. Try white papers especially Tcl: An Embeddable Command Language .

Erlang? (none / 0) (#35)
by scheme on Sat Jan 20, 2001 at 12:12:33 AM EST

How about looking at Erlang? Ericsson and Bluetail use it extensively to program their telecom hardware.


"Put your hand on a hot stove for a minute, and it seems like an hour. Sit with a pretty girl for an hour, and it seems like a minute. THAT'S relativity." --Albert Einstein


Rearchitect. (none / 0) (#37)
by Chiron on Sat Jan 20, 2001 at 03:06:18 PM EST

I know, this is exactly the answer you don't want to hear, and the one you've probably been considering, but you may want to consider redesigning your application. Very few of the rich, high level languages lack side effects that make coarse grained locks necessary on the interpreter.

Let me warn you, I am a Python zealot. My suggestion would be to redesign your app so whatever time-critical functions and various C and C++ APIs that you cannout live without are made available to a Python-based framework. Never underestimate a high level language with judicious use of profiling and a willingness to expand.

Considering the amount of threads your app seems to require, I would suggest the use of Stackless Python and the Microthreading library, or a move to a language better suited to concurrent design. If you can wrap your head around Functional Programming, I would suggest Erlang, which is also easy to extend, and has excellent concurrency support.

Best of luck!

Need help choosing an embedded language | 37 comments (32 topical, 5 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!