Welcome, Guest. Please login or register.

Login with username, password and session length

 
Advanced search

1411665 Posts in 69396 Topics- by 58452 Members - Latest Member: Monkey Nuts

May 16, 2024, 06:32:32 AM

Need hosting? Check out Digital Ocean
(more details in this thread)
TIGSource ForumsDeveloperTechnical (Moderator: ThemsAllTook)Embeddable Scripting Languages
Pages: [1] 2
Print
Author Topic: Embeddable Scripting Languages  (Read 5668 times)
Garthy
Level 9
****


Quack, verily


View Profile WWW
« on: December 07, 2009, 07:54:54 PM »

I seem to have been fighting this problem for years. I'm wondering if anyone can help. I'm trying to find a embeddable scripting language that will allow me to do this:

In a C/C++ program, set up a scripting environment, load a script (from a file), push some local C/C++ functions into the environment, and then call the script, which may contain calls to the functions I pushed into the environment. Basically:

A (C++) calls B (scripting language) which calls a function in A (C++).

An opaque type needs to be passed in through the chain, and it needs to be able to handle simple things, such as passing around integers, floats, and strings.

Note that being able to include C/C++ code in a scripting language as an extension is *not* enough. I know that a lot of things can do this- it's not what I'm trying to do, and not what I want.

Now, there's a bunch of languages that *might* do it, and tools that *might* help. I'm after something sufficiently advanced that it *specifically* presents a clear and easy-to-use example of this sort of thing in their documentation.

So far, I've got this sort of thing working with LUA, but I'm after other options. Unfortunately, for most things that claim that they can do this, there's minimal documentation and rarely an example of how to do it. Most of the examples end up focusing on extending the scripting language, rather than embedding it in an existing program. I end up sinking a whole bunch of time into trying to figure it out, then find out that I can't *really* do it, the minimal doco that hinted it was possible was actually wrong, or there's some nasty catch that is being glossed over.

Here is a concrete example of what I'd like to do:

*** foo.cpp:

Code:
class Bar
{
  public:

    Bar();
    int foo(int b, int c);
    
    int run0(int e);
    int run1(int e, int f);

    ScriptingEnvironment env;
    int d;

  (other, really complex definitions)
};

// Just an adapter to turn the opaque type back into a class, then
// call a method. I'm assuming the environment handles opaque types with
// (void *).
static int call_foo(void *a, int b, int c)
{
  return ((Bar *)a)->foo(b, c);
}

// The function that may be called from the script after passing through the
// adapter.
int Bar::foo(int b, int c)
{
  return somecomplexfunction(b, c, d);
}

void Bar::Bar()
{
  // Create the environment.
  env = ScriptingEnvironment::create();
  // Add our local adapter function "call_foo", accessible as "foo" from
  // the script.
  env.add("foo", call_foo, "pii") // Pointer, int, int.
  // Load the script from an external file. I don't mind if I have to load
  // first then add funcs, or add funcs then load- works for me.
  env.load("somescript.ext");
  // Any explicit exporting would be done here, if needed.
  env.export("somefunc", "pi");
  env.export("otherfunc", "pii");
}

int Bar::run0(int e)
{
  // Setting "d", which we end up using in "foo" if it is called, but not
  // otherwise.
  d = somemagicfunc(e);
  // We pass in "this" as an opaque type, so that if it calls "call_foo",
  // we can convert it back to the class via the adapter.
  return env.call("somefunc", "pi", this, e); // Pointer, int.
}

int Bar::run1(int e, int f)
{
  d = somemagicfunc(e+f);
  return env.call("otherfunc", "pii", this, e, f); // Pointer, int, int.
}

int main()
{
  // Construct a "Bar" named "bar", which will load the script and do the
  // needed setup.
  Bar bar;
  // Make multiple calls to the loaded script and environment.
  cerr << bar.run0(4) << "\n";
  cerr << bar.run0(5) << "\n";
  cerr << bar.run0(6) << "\n";
  cerr << bar.run1(4) << "\n";
  cerr << bar.run1(5) << "\n";
  cerr << bar.run1(6) << "\n";
}

*** somescript.ext:

Code:
#NB: "foo" is impemented back in the original C++.

# This is called from the C/C++ side.
def somefunc(a, e)
  return foo(a, e) + foo(a, 5)
end

# This is called from the C/C++ side.
def otherfunc(a, e, d)
  return foo(a, e) + foo(a, 6) + foo(a, d)
end

..............


So the question is: Is anyone aware of an embeddable scripting language (apart from LUA) that *can* do this, has personally done it themselves, knows the official documentation has *actual* easy-to-use examples that show the officially-supported way of doing it, and can also recommend it?
« Last Edit: December 08, 2009, 03:00:43 PM by Garthy » Logged
David Pittman
Level 2
**


MAEK GAEM


View Profile WWW
« Reply #1 on: December 08, 2009, 02:26:00 PM »

I too am curious if people have succeeded with this (although I assume most scripting languages have some way to do this because the sanest way to handle basic system libraries is handing them off to the default C implementations).

I wanted to do this Python once, and it seems to be not too different from Lua (except for needing to link with the Python interpreter), from a glance at the documentation and example. But I can't actually recommend it because I ended up deciding I didn't really need a scripting language and never tried it out.
Logged

Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #2 on: December 08, 2009, 02:33:23 PM »

I was about to comment that the linked example was extending rather than embedding- and then I scrolled down to section 1.6. I'm having a read through now.

EDIT: Turns out it was just how to call a Python function from C that was already called by Python. So it still seems to be extending rather then embedding- I'm happy to be shown to be wrong though.

EDIT 2: Seems to be a brief mention of embedding on that page starting with the text "When embedding Python", along with a brief example of initialisation, and a reference to some external code that may also be an example. Still lacking a concrete example of the whole thing though, which is frustrating. It's a shame that embedding frequently is approached as an afterthought with these sorts of things. Has anyone succeeded in embedding Python directly in this way?
« Last Edit: December 08, 2009, 02:48:28 PM by Garthy » Logged
Average Software
Level 10
*****

Fleeing all W'rkncacnter


View Profile WWW
« Reply #3 on: December 08, 2009, 02:52:00 PM »

Just curious, but what compiler are you using that allows you to declare main as static?  That's very much illegal, and should result in a program that can't run, since the linker should be unable to find the entry point.
Logged



What would John Carmack do?
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #4 on: December 08, 2009, 03:01:35 PM »

Just curious, but what compiler are you using that allows you to declare main as static?  That's very much illegal, and should result in a program that can't run, since the linker should be unable to find the entry point.

Fixed. Consider the above pseudo-code, to illustrate the thing I am trying to achieve.

Logged
Ben Kuhn
Level 0
***


View Profile
« Reply #5 on: December 08, 2009, 04:59:33 PM »

Swig plus the Python/C API will solve your problem.

The way that extending Python works is that a module will call the C API function Py_InitModule("foo", fooFunctionTable) with the relevant arguments. Through the magic of dynamic linking, this will add the method table of the module to whatever Python interpreter is lying in memory - whether it is the main Python executable or an embedded interpreter.

Now, the way that Python modules make this happen is that they have a special function with the signature PyMODINIT_FUNC init_foo(void). Inside this function, the code that extends Python with wrapper functions/variables is called. When you say 'import foo' in Python, Python looks for a C library called 'foo.so' or 'foo.dll'. Then it looks inside that library for the magic function named above, and calls it.

But if you're embedding the Python DLL into your own executable, you have the privilege of doing whatever the hell you want with the C API before (or after) you pass control to Python. So you can simply call that same function inside your main function (after you've initialized Python, of course), and those same C functions will be accessible from Python. This way you don't have to bother with vomiting your module file onto a search path, etc. In fact, your "module" doesn't even have to be in a separate library from your main code - it will work fine either way. You can call init_foo() from wherever you want and it will always publish the module "foo" into the global Python namespace.

Now, after you've "imported" your "module", either by telling Python "import foo" or by calling init_foo() from C code, all Python code that the Python DLL runs will be able to call functions like "foo.baz(bar, 6)" that are defined in C code. To run Python code and communicate with Python (i.e. get and set variables, call functions, whatever), you can use the Python/C API.

Further reading:
How to embed Python: http://docs.python.org/extending/embedding.html - should tell you everything important about how to communicate with Python
Python/C API reference: http://docs.python.org/c-api/index.html - if there's something the embedding guide doesn't tell you, look here

I feel like I may have missed the point of your question. But you can definitely do what you're trying to with Python + SWIG.

EDIT: Passing opaque types from toplevel to Python to SWIG-wrapped function is a bit trickier, but still doable. You should dig around in the code that SWIG generates and look at/read up on typemaps - SWIG should have a function in there somewhere that will take a pointer of whatever type and turn it into the right kind of PyObject.
« Last Edit: December 08, 2009, 05:03:43 PM by Ben Kuhn » Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #6 on: December 08, 2009, 05:28:18 PM »

Thankyou Ben for the detailed description- it has helped immensely. Smiley And to top it off, the first link you've provided is PURE GOLD. Smiley It's basically a howto of *exactly* what I want to do, and the specific A->B->A case I was after is documented in section 5.4 of the link you've provided. Plus, there are actual code examples. My problem was effectively that I was looking in the wrong part of the manual- it *was* documented properly, I just failed to find it...

The description you've provided has also helped with respect to how I can get some of the local functions in there as well. If I have a great number of them, and my understanding is correct from what you have said, I can use SWIG to quickly get them all in there. Basically the output of SWIG should generate the init_* calls that I need to add, or something very close to that anyway. I'm not sure how I'll go with opaque types, but my guess is that second link is going to give me a hint, and worst-case I'll just pass in integers and map it back to pointers after (which *will* work, is just messy). From the sounds of things, starting off getting the basics going directly with the Python API is going to be the best thing to begin with, and then I can check out SWIG to simplify function wrapping, and I might give Boost::Python a miss for now.

Okay- this is really good news. Quite exciting actually. Smiley I'm not used to embedded interfaces actually having decent documentation as a general rule. Smiley Ben, thankyou so much for persisting! (for all others: Ben also posted in my original thread trying to help).

With any luck I'll probably be needing to brush up on my Python again. Wink

Now, the last question would be: Has anyone actually done this (with Python specifically), and if so, how did you find it? Any snags or catches to be aware of?
Logged
Ben Kuhn
Level 0
***


View Profile
« Reply #7 on: December 08, 2009, 05:45:05 PM »

Thankyou Ben for the detailed description- it has helped immensely. Smiley And to top it off, the first link you've provided is PURE GOLD. Smiley It's basically a howto of *exactly* what I want to do, and the specific A->B->A case I was after is documented in section 5.4 of the link you've provided. Plus, there are actual code examples. My problem was effectively that I was looking in the wrong part of the manual- it *was* documented properly, I just failed to find it...

The description you've provided has also helped with respect to how I can get some of the local functions in there as well. If I have a great number of them, and my understanding is correct from what you have said, I can use SWIG to quickly get them all in there. Basically the output of SWIG should generate the init_* calls that I need to add, or something very close to that anyway.

No problem, happy to help. All the interlocking parts can be hard to grok at first, so I'm glad that one more person doesn't have to deal with that.

I would suggest that if the second link doesn't give you any help, you dig around in the code that SWIG barfs out and see how it wraps a C pointer into a PyObject. I think that it may generate functions that will do that: for example, if you have
Code:
//foo.cpp
class Foo {
//complicated stuff
};
//foo.i
class Foo {};
Then SWIG will spit out a function named something like
Code:
PyObject * wrapFooPointer(Foo* f);
and you could do
Code:
//equivalent to globals["argle"]=Foo(whatever) in python
PyDict_SetItemString(PyEval_GetGlobals(), "argle", wrapFooPointer(argle));
in C to get what you're looking for. (Names subject to change, of course, and you might have to muck about with refcounting.)
Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #8 on: December 08, 2009, 05:52:26 PM »

I would suggest that if the second link doesn't give you any help, you dig around in the code that SWIG barfs out and see how it wraps a C pointer into a PyObject. I think that it may generate functions that will do that

That's a neat idea. I could probably get the appropriate unwrap code in a similar way too. Smiley
Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #9 on: February 08, 2010, 04:05:49 PM »

Just posting to provide an update on this.

As of a couple of days ago, the time finally arrived where I had to make a call on my embedded scripting environment, and I gave Python a shot based on this thread and in particular Ben's helpful feedback.

It has turned out reasonably well thus far. There were some undocumented frustrations (what is the best way to get feedback back to the people who write the embedded docs?), and I almost gave up on it due to one thing in particular [1], but in the end, I was able to get a basic test case going, that did the A->B->A thing I needed and described. Since then, I've developed a light wrapper to handle some of the more tedious aspects of embedding for me, and got it going in my project under both Linux and Windows, where it works equally well.

It is of course, still very much an internal "test case" (that just runs a few dummy operations), and I won't know how it works in "real life" until I've got it driving the bits of the code that need the scripting, which doesn't fully exist yet. :} But from what I've seen, it looks very promising. My plan at this point is to develop the main code that requires scripting, and increase the scope of what is covered as much as is reliably possible.

I generally like the API- decent reference documentation too. And as for the quality and ease of use of the Python language itself- there are few equals.

Anyway, I'll share a few tips and workarounds in case someone else tries going along the same path. Hopefully I can save someone the few frustrating hours I spent discovering these things the hard way. Note that these are in addition to the documentation, which is reasonable, but incomplete (IMHO):

- (This cost me some very frustrating hours to find out) To load a module from a file, it either needs to be in the PYTHONPATH environment variable already (!), or in sys.path. To, say, load from the current working directory, issuing these commands right after Py_Initialize() will do the trick:
  - PyRun_SimpleString("import sys");
  - PyRun_SimpleString("sys.path.append('.')");
  - IIRC, the file should then be called "foo.py" if you wish to import it as "foo", although I haven't got an easy way to doublecheck that at present.

- (This took some experimenting) If you want to load a Python script from memory (or a string), without it ever ending up on the filesystem, you need these two calls: Py_CompileString(), then PyImport_ExecCodeModule(), using the result of the former.

- (I was lucky on this one) You probably want to issue: "Py_NoSiteFlag = 1;" before Py_Initialize(). If you do not, it'll search for a suitable "site.py" file, which you probably won't always have, unless you include one with your app. You wont always notice this, since it tends to find it on a development environment.

- PyRun_SimpleString() is your new best friend. It runs a single command in the current Python environment, and the results are persistent.

- If you load a module with the same name into the current environment, it will effectively overlay the functions in the same space. However, if you have already done an "import sys" via PyRun_SimpleString(), and then issue a "del sys.modules[\"modname\"]", you'll get a clean import. Note that there are issues with lingering non-visible modules that occur if the module was in use somehow at the time.

- Inside extended functions in an embedded environment, the "self" object may be NULL, and the "args" object is generally a tuple, that you can confirm with PyTuple_Check(), get the size of with PyTuple_Size(), and raid with PyTuple_GetItem().

- If you change the library name under Windows, also update the mention in the pyconfig.h file, or you'll get linker errors. Also, don't forget to copy the pyconfig.h file into the right place when building under Windows.

Hope this saves others a bit of time. Smiley

[1] Embedded Python by default will import modules based on the PYTHONPATH environment variable, which doesn't include the pwd and generally isn't set to what you want when embedding, and there's no hint that this is the case- this makes loading your own test module difficult unless you guessed the issue ahead of time.
« Last Edit: February 08, 2010, 04:10:52 PM by Garthy » Logged
Parthon
Level 1
*


View Profile
« Reply #10 on: February 08, 2010, 06:08:04 PM »

I managed to get Python embedded successfully through C++ a while ago.

Problems I found with it:

Every function in C++ that needs to be called from Python needs to be initialised. Eventually the script class that handles it gets a little complicated from all of the different types of calls.

The other thing I couldn't solve: I could call a function in Python, but then when it returned I'd lose all the data in Python. I solved this by just staying in Python the whole time and calling needed functions in C++ from Python. It ended up being like an extension, except the environment and engine set up in C++ before the Python was called make embedding better than extending. If the Python values were persistant then I wouldn't have to do it that way, but it worked.
Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #11 on: February 08, 2010, 07:50:34 PM »

I managed to get Python embedded successfully through C++ a while ago.

Problems I found with it:

Every function in C++ that needs to be called from Python needs to be initialised. Eventually the script class that handles it gets a little complicated from all of the different types of calls.

I'm presently exposing such functions via Py_InitModule(), a table of calls, and in the script it imports that "virtual" module, and makes the calls. Did you do something similar? If so, is there a gotchya that I'm about to run into? Would love to know about it in advance if I can. Smiley

I'm also not sure what you mean when you say that every function needs to be initialised- apart from my Py_InitModule() call, I don't actually do any other prep for the callback/A->B->A-style functions.

The other thing I couldn't solve: I could call a function in Python, but then when it returned I'd lose all the data in Python. I solved this by just staying in Python the whole time and calling needed functions in C++ from Python. It ended up being like an extension, except the environment and engine set up in C++ before the Python was called make embedding better than extending. If the Python values were persistant then I wouldn't have to do it that way, but it worked.

If anything, I've found Python was too keen to hold onto information than too willing to give it up. Wink Could it be related to not bumping up the reference count on certain objects, causing the GC to go crazy on it and discard it? Just a guess. Either way, I'll keep an eye out for things that match these symptoms. Smiley

Apart from the gotchyas mentioned, would you say that using Python in an embedded sense was a decision you were happy with, or something you ended up regretting?
Logged
Parthon
Level 1
*


View Profile
« Reply #12 on: February 08, 2010, 08:16:55 PM »

I used InitModule with a table of calls too. Just the table started getting long. Wink It wasn't really a problem with python/c++, I think the problem was with my implimentation. I'd do it differently the next time around.

As for the missing data, I think everything got lost when the function returned back to c++ because it went out of scope. I'd need to learn more about Python to stop that from happening. Wink

I didn't regret it, because I learnt a lot. The larger problem was that I found out that I didn't know enough about Python to actually impliment anything worthwhile. I'd have to learn Python from the ground up as well as making a new game as well as working with Python/C++ at the same time. It was chosen because it meant the game could be edited without having to recompile.

Recently I dropped the Python and went back to pure C++. I'll probably reinvestigate the use of Python in the future. It would be handy in a scripted game, but I never got that far.  Shrug
Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #13 on: February 08, 2010, 08:43:16 PM »

Ah, cheers, thanks for that. Smiley

I used InitModule with a table of calls too. Just the table started getting long. Wink It wasn't really a problem with python/c++, I think the problem was with my implimentation. I'd do it differently the next time around.

As for the missing data, I think everything got lost when the function returned back to c++ because it went out of scope. I'd need to learn more about Python to stop that from happening. Wink

When wrapping some parts of Python I considered doing some sort of clever wrapper to manage those tables, but decided against it as the implementation I'm working on will likely only have a small number of calls (maybe a dozen), and they'll generally only change statically. Of course, experience could prove my guess wrong. :} I think I understand what you mean with respect to those tables now- each time you changed them, you'd need to re-init the same module, and you'd have hassles if anything was using that module at the time.

I didn't regret it, because I learnt a lot. The larger problem was that I found out that I didn't know enough about Python to actually impliment anything worthwhile. I'd have to learn Python from the ground up as well as making a new game as well as working with Python/C++ at the same time. It was chosen because it meant the game could be edited without having to recompile.

Sounds good. Smiley My Python is very rusty, but I imagine I'll be learning it again in very short order. Smiley The sort of stuff I'll be doing is quite well suited to scripting: too dynamic for it to be conveniently built into an executable, but too complex to be data-driven. And, once it's in there, I'll probably find a dozen other uses for which a scripting language is more maintainable than a more detail-oriented language such as C++. :}
Logged
encoder
Level 0
**


View Profile WWW
« Reply #14 on: February 09, 2010, 01:19:21 AM »

there is a chance flash player gets open source this year. now imagine that! not only you have a powerful scripting language but you also have control over it's graphics engine. what 2D is concerned, your set. and i'm sure with some modifications you can directly inject AS3 and run it. this was removed due to security reasons, but it's still there in AS2 (run at script level). anyway, a swf is the script and graphics resources mainly, so there should not be any problem to make a console out of it.

i'm beginning to feel like adobe's paladin.. good. Kiss
Logged

god is an instance of a class. who is the coder?
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #15 on: February 09, 2010, 02:43:01 AM »

there is a chance flash player gets open source this year. now imagine that! not only you have a powerful scripting language but you also have control over it's graphics engine. what 2D is concerned, your set. and i'm sure with some modifications you can directly inject AS3 and run it. this was removed due to security reasons, but it's still there in AS2 (run at script level). anyway, a swf is the script and graphics resources mainly, so there should not be any problem to make a console out of it.

i'm beginning to feel like adobe's paladin.. good. Kiss

I have to admit that this post has completely and utterly baffled me. Personally, why do you feel that Actionscript would be a good solution to this problem?
Logged
salade
Level 4
****



View Profile
« Reply #16 on: February 09, 2010, 03:13:37 PM »

I think he is trying to say that having actionscript will solve your problem if flash goes open source; still nonsensical though.

Quick question - why did you ditch LUA? I was thinking of using it primarily since it is able to do exactly what you are looking for.

Nevermind, got mixed up, thought you meant that you were looking to call functions in script not that you wanted to do more.

« Last Edit: February 09, 2010, 03:17:20 PM by salade » Logged
Garthy
Level 9
****


Quack, verily


View Profile WWW
« Reply #17 on: February 09, 2010, 03:32:08 PM »

Quick question - why did you ditch LUA? I was thinking of using it primarily since it is able to do exactly what you are looking for.

Nevermind, got mixed up, thought you meant that you were looking to call functions in script not that you wanted to do more.

Actually, you're quite right- I am looking to ditch LUA. Or, more accurately, I used LUA for embedded scripting in my last project, found a few aspects er... not to my taste, and was looking to try out other options for my next one. I would highly recommend investigating it and experimenting with it as a potential candidate. It it basically my fallback option if I can't get a more preferred language to work.

On that front, as of today, I've now got Python driving one of the two key parts of my project that need scripting. Thus far, pretty stable. At this point I'd offer even odds that it will be the final thing that I use.
Logged
Bad Sector
Level 3
***


View Profile WWW
« Reply #18 on: February 09, 2010, 04:43:46 PM »

I'll propose my Minimal scripting language. Its very simple and a good starting point for making your own scripting language (the code is hopefully easy to follow and its just a pair of .c and .h files). Personally i use it as a starting point for game/engine-specific scripting languages or configuration files so the Minimal lang itself doesn't have much stuff.

Another option is AngelScript which can use C++ functions and classes directly and i think even extend them.
Logged

~bs~
westquote
Level 0
**


I make games in Philly. How rare!


View Profile WWW
« Reply #19 on: February 09, 2010, 06:10:21 PM »

Actually, you're quite right- I am looking to ditch LUA. Or, more accurately, I used LUA for embedded scripting in my last project, found a few aspects er... not to my taste, and was looking to try out other options for my next one.

What aspects in particular rubbed you the wrong way?  I'd be interested to hear more about your experiences integrating and using it in your game.  Such accounts are pretty hard to come by, and I'm quite curious to hear how things play out for other developers.
Logged

Twitter: @westquote - Webpage: Final Form Games
Pages: [1] 2
Print
Jump to:  

Theme orange-lt created by panic