ExtUtils::xsbuilder.osc2002.3pm

Langue: en

Autres versions - même langue

Version: 2002-08-05 (fedora - 01/12/10)

Section: 3 (Bibliothèques de fonctions)

Sommaire

XSBuilder - Autogenerating XS-glue Code

O'Reilly OpenSource Convention 2002

Gerald Richter

ecos gmbh

http://www.ecos.de

XSBuilder - What is it?

It's purpose is to automaticly generate a Perl interface to C Code

Solves the problem, that the Perl interface is not always uptodate with the C interface

Saves a lot of Copy&Paste work

Systematical changes have to be done only once

For example changes in the memory management of strings.

Is part of mod_perl 2.0 build system

Most code is developed by Doug MacEachern. Additionaly I have
abstracted the code from mod_perl so it's useable for any C code
added new features like callbacks and the ability to parse comments
Replaced C::Scan with a Parse::RecDescent to be platform and compiler independend

Goal is to replace the current mod_perl XS generation code with XSBuilder

Inline versus XSBuilder

Inline: embed C-Code into Perl
XSBuilder: Create interface for existing C-libraries/applicationen

XSBuilder - What does it do?

Create Perl functions/methods for every C function

The function can be assigned to different packages, also automaticly by inspecting the first parameter

Create a Perl class for every C structure

Every element of structure becomes a Perl method to get/set it's value. The object can be either a scalar reference (used by mod_perl) or a reference to a hash (use by Embperl), which allows to store extra data by the Perl code into this hash.

Create glue code to handle callbacks

There several sorts of callback, not all are implemented right now

Create Perl constant subs

Coming soon...

XSBuilder - How does it work?

Parse the C header files

Extract
Functions, their arguments and return types
Structures and it's members
Constants
Callbacks

and create four tables which contains the results

Create the XS code

Input is
The source tables
Mapfiles which contains the mapping from C to Perl
Addtional C and Perl code that can be used to customize the interface

Output is

The XS files (one form every generated class)
Makefile.PL for every class
pm files

Parse the source

Create your own ParseSource class and override methods...

     package Apache::DAV::ParseSource;
 
     use strict;
     use vars qw{@ISA $VERSION} ;
     use ExtUtils::XSBuilder::ParseSource  ;
 
     @ISA = ('ExtUtils::XSBuilder::ParseSource') ;
     $VERSION = '0.01';
 
     my $dav_dir = 'C:\perl\msrc\cvs\mod_dav' ;
     my $ap_dir  = 'c:\programme\apache group\apache' ;
 
     # ============================================================================
     sub find_includes {
         my $self = shift;
         return $self->{includes} if $self->{includes};
         my @includes = ("$ap_dir/include/ap_alloc.h", "$dav_dir/mod_dav.h") ;
         return $self->{includes} = $self -> sort_includes (\@includes) ;
         }
 
     # ============================================================================
     sub package     { 'Apache::DAV' } 
 
     # ============================================================================
     sub preprocess {
         my $self     = shift ;
         $_[0] =~ s/(?:API_EXPORT)(?:_NONSTD)?\s*\(\s*(.*?)\s*\)/$1/g ;
     }
 
 
     1;

...run it

     use FindBin ;
     use lib ($FindBin::Bin) ;
 
     require ParseSource ; 
 
     Apache::DAV::ParseSource -> run ;

...and you get

     C:\perl\msrc\davint>perl xsbuilder\source_scan.pl
     Will use mod_dav in C:\perl\msrc\cvs\mod_dav
     Will use Apache in c:\programme\apache group\apache
     Initialize parser
     scan c:\programme\apache group\apache/include/ap_alloc.h ...
     constant: APACHE_ALLOC_H
     func:     ap_init_alloc
     func:     ap_cleanup_alloc
     func:     ap_make_sub_pool
     func:     ap_destroy_pool
     constant: ap_pool_join
     func:     ap_pool_join
     func:     ap_find_pool
     func:     ap_pool_is_ancestor
     func:     ap_clear_pool
     func:     ap_cleanup_for_exec
     func:     ap_palloc
     func:     ap_pcalloc
     func:     ap_pstrdup
     func:     ap_pstrndup
     func:     ap_pstrcat
     func:     ap_pvsprintf
       valuefield: ap_pool * : pool
       valuefield: int : elt_size
       valuefield: int : nelts
       valuefield: int : nalloc
       valuefield: char * : elts
     struct:    (type=array_header)
     ...

The result is stored in four tables

xsbuilder/tables/Apache/DAV/FuntionTable.pm
Contains all function, it arguments and comments
xsbuilder/tables/Apache/DAV/ConstantTable.pm
Contains all constants
xsbuilder/tables/Apache/DAV/StructureTable.pm
Contains all structures, it's members and their comments
xsbuilder/tables/Apache/DAV/CallbackTable.pm
Contains all callback function definitions

Create the map files

Mapfiles are used to tell XSBuilder how C datatypes, structures and function aruments should be mapped into Perl ones.

Create your own WrapXS class and override methods

     package Apache::DAV::WrapXS ;
     use strict;
     use vars qw{@ISA $VERSION} ;
 
     use ExtUtils::XSBuilder::WrapXS ;
 
     @ISA = ('ExtUtils::XSBuilder::WrapXS') ;
     $VERSION = '0.01';
 
     # ============================================================================
     sub new_parsesource  { [ Apache::DAV::ParseSource->new ] }
 
     # ============================================================================
     sub my_xs_prefix  { 'davxs_' }
 
     # ============================================================================
     sub h_filename_prefix  { 'moddav_xs_' }
 
     # ============================================================================
     sub xs_includes {
         my $self = shift ;
         my $i = $self -> SUPER::xs_includes ;
         my @i = grep (!/ap_alloc/, @$i) ;
         return \@i ;
     }

XSBuilder can create/update initial maps for you

     use FindBin ;
     use lib ($FindBin::Bin) ;
 
     require ParseSource ; 
     require WrapXS ; 
 
     Apache::DAV::WrapXS->checkmaps (' ');

run it

     C:\perl\msrc\davint>perl xsbuilder\xs_check.pl
     Will use mod_dav in C:\perl\msrc\cvs\mod_dav
     Will use Apache in c:\programme\apache group\apache
     Parse xsbuilder\maps/_types.map...
     WARNING: No *_function.map file found in xsbuilder\maps
     WARNING: No *_callback.map file found in xsbuilder\maps
     WARNING: No *_structure.map file found in xsbuilder\maps
     Write xsbuilder\maps/new_function.map...
     Write xsbuilder\maps/new_callback.map...
     Write xsbuilder\maps/new_structure.map...
     Write xsbuilder\maps/new_type.map...

Now we have four map files

new_types.map
Contains the mapping from C type to Perl classes
new_functions.map
Contains the mapping form C functions to Perl functions. Can be used to reorder arguments, tell XSBuilder which arguments are actualy return values and in which Perl package the function will be created.
new_structures.map
Contains the mapping from C structures to Perl classes and defines for which members a access methods should be created. You can also specify if you want a "new" method for the class.
new_callbacks.map
Contains the mapping form C callback functions to Perl callback functions. Can be used to reorder arguments, tell XSBuilder which arguments are actualy return values and in which Perl package the function will be created.

It's a good idea to rename the prefix from "new_" to something unique, here we use "dav"

Everytime you rerun checkmaps, XSBuilder will create new_* files with the items that are not already part of the other maps.

Next step is to customize the maps...

type map

autogenerated dav_type.map

     DIR |
     FILE        |
     HANDLE      |
     array_header        |
     dav_buffer  |
     dav_dyn_context     |
     dav_dyn_hooks       |
     dav_dyn_module      |
     dav_dyn_provider    |
     dav_error   |
     dav_get_props_result        |
     dav_hooks_liveprop  |
     dav_hooks_locks     |
     dav_hooks_repository        |
     dav_if_header       |
     dav_if_state_type   |
     ...

Add Perl classes

     struct array_header         | Apache::Array
     struct dav_buffer           |
     struct dav_datum            | Apache::DAV::Datum
     struct dav_dyn_context      | Apache::DAV::DynContext
     struct dav_dyn_hooks        | Apache::DAV::DynHooks
     struct dav_dyn_module       | Apache::DAV::DynModule
     struct dav_dyn_provider     | Apache::DAV::DynProvider
     struct dav_error            | Apache::DAV::Error
     struct dav_get_props_result | Apache::DAV::PropsResult
     struct dav_hooks_db         | Apache::DAV::HooksDb
     struct dav_hooks_liveprop   | Apache::DAV::HooksLiveprop
     struct dav_hooks_locks      | Apache::DAV::HooksLocks
     struct dav_hooks_repository | Apache::DAV::HooksRepository
     struct dav_hooks_vsn        | 
     struct dav_if_header        | Apache::DAV::IfHeader
     struct dav_if_state_list    | Apache::DAV::StateList
     ...
     struct pool                 | Apache::Pool
     struct request_rec          | Apache::
     struct server_rec           | Apache::Server
     ...

Defines the mapping from C datatypes to Perl datatypes and classes and tells XSBuilder which datatype are (like) structures

function map

Function map defines the mapping from C functions arguments to Perl arguments
Tell XSBuilder where to place functions and which prefix to strip
     MODULE=Apache::DAV   PACKAGE=guess  PREFIX=dav_
Simple entries in the function map will be mapped 1:1 from C to Perl
      dav_add_response
      dav_buffer_append
      dav_buffer_init
      dav_buffer_place
      dav_buffer_place_mem
      dav_check_bufsize
      dav_close_propdb
      dav_collect_liveprop_uris
      dav_dyn_module_add
      dav_empty_elem
     ...
The following map file entry tells XSBuilder that the value of res_p should be returned
      dav_get_resource           | | r, <res_p

The C function

     int dav_get_resource(request_rec *r, dav_resource **res_p) ;

How to call it from Perl

     # get a sub request of the actual document
     $subr = $r -> lookup_uri($uri);
 
     # get a mod_dav resource object
     my ($err, $resource) =  $subr->get_resource;
You can let XSBuilder insert your custom code, for the interface
If you call "get_props" from Perl "glue_dav_get_props" will be called, which can adjust the arguments and return types as necessary. The actual code for "glue_dav_get_props" will be taken from separate include file. 
      dav_get_props      | glue_
      dav_get_allprops   | glue_

xsinclude\Apache\DAV\PropResult\Apache__DAV__PropResults.h

     dav_get_props_result * dav_glue_get_props(dav_propdb * db, dav_xml_doc *doc)
         {
         dav_get_props_result * result = (dav_get_props_result *)ap_palloc (db -> p, sizeof (dav_get_props_result)) ;
         *result = dav_get_props(db, doc) ;
         return result ;
         }
Arguments can be replaced
     MODULE=Apache::Array   PACKAGE=Apache::Array PREFIX=ap_
 
      ap_make_array      | ap_make_array(r->pool, nelts, elt_size) | request_rec *:r, nelts, elt_size

ap_make_array requires a pool a it's first parameter, we pass the request_rec from Perl and XSBuilder will take the pool from the request_rec.

structure map

      MALLOC=strdup:$dest = ($type)strdup($src)
      FREE=strdup:free($src)
 
      <dav_xml_elem>
        name
        ns
        lang
        first_cdata
        following_cdata
        parent
        next
        first_child
        attr
        last_child
        ns_scope
        propid
        provider
        ns_map
        new
      </dav_xml_elem>
 
      MALLOC=strdup:$dest = ($type)ap_pstrdup(obj -> pool, $src)
 
      <array_header>
        pool
        elt_size
        nelts
        nalloc
        elts
        private
      </array_header>

Create a accessor functions for every element and, if requested, a new method

         $setprop = Apache::DAV::XMLElem -> new ({name => 'prop'}) ;
         $elem    = Apache::DAV::XMLElem -> new ({name => $name, ns => $namespaces}) ; 
         $setprop -> first_child($elem) ;
         $first = $setprop -> first_child ;

some datatypes, like strings, requires dynamic allocated memory

From _types.map,which conatins a set of standard types 
     int             | IV
     int *           | UNDEFINED
     unsigned int    | UV
     signed int      | IV
     long            | IV
     long int        | IV
     unsigned long   | UV
     unsigned        | UV
 
     char *                   | PV           | | | strdup
     const char *             | PV           | | | strdup
     const char **            | UNDEFINED    
     char const *             | PV           | | | strdup
     unsigned char *          | PV           | | | strdup
     const unsigned char *    | PV           | | | strdup
     ...

callback maps

Callback maps have the same options a function maps 
     # dav_hooks_db -> open
      
      dav_error *(*)(pool * p,const dav_resource * resource,int ro,dav_db * * pdb)  | p, resource, ro=0, <pdb 
     
     
     # dav_hooks_lock -> has_locks
     
      dav_error *(*)(dav_lockdb * lockdb,const dav_resource * resource,int * locks_present)

Generate the XS files

     use FindBin ;
     use lib ($FindBin::Bin) ;
 
     require ParseSource ; 
     require WrapXS ; 
 
     Apache::DAV::WrapXS->run;

...and run...

     C:\perl\msrc\davint>perl xsbuilder\xs_generate.pl
     Will use mod_dav in C:\perl\msrc\cvs\mod_dav
     Will use Apache in c:\programme\apache group\apache
     Parse xsbuilder\maps/_types.map...
     Parse xsbuilder\maps/dav_type.map...
     mkdir xs
     writing...xs//typemap
     Parse xsbuilder\maps/dav_functions.map...
     WARNING: Cannot map type int(*)(void * ,const char * ,const char * ) for function ap_table_do
     WARNING: Cannot map type dav_buffer * for function dav_buffer_append
     WARNING: Cannot map type dav_buffer * for function dav_buffer_init
     WARNING: Cannot map type dav_buffer * for function dav_buffer_place
     WARNING: Cannot map type dav_buffer * for function dav_buffer_place_mem
     WARNING: Cannot map type dav_buffer * for function dav_check_bufsize
     WARNING: Cannot map return type int * for function dav_collect_liveprop_uris
     WARNING: Cannot map type dav_resource * * for function dav_ensure_resource_writable
     WARNING: Cannot map type dav_buffer * for function dav_lock_get_activelock
     WARNING: Cannot map type dav_buffer * for function dav_set_bufsize
     WARNING: Cannot map type int * for function dav_xml2text
     struct array_header...
     Parse xsbuilder\maps/dav_structure.map...
          elt_size...
          nelts...
          nalloc...
          elts...
     struct dav_buffer...
     struct dav_datum...
          dptr...
          dsize...
     struct dav_dyn_context...

Makefile.PL

We need create a top level Makefile.PL

     use ExtUtils::MakeMaker ();
     
     my $apdir  = '/path/to/apache';
     my $davdir = '/path/to/moddav';
 
     %MMARGS = (
                'INC'    => "-I\"$davdir\" -I\"$apdir/include\" -I\"$apdir/os/unix\" -I\"$dir/xs\"  -I\"$dir/xsinclude\"",
                ) ;
 
 
     open FH, ">xs/mmargs.pl" or die "Cannot open xs/mmargs.pl ($!)" ;
     print FH Data::Dumper -> Dump ([\%MMARGS], ['MMARGS']) ;
     close FH ;
 
 
     ExtUtils::MakeMaker::WriteMakefile(
         'NAME'    => 'Apache::DAV',
         'VERSION' => '0.13',
         %MMARGS,
     );

Makefile.PL's for all class are generated automaticly

     C:\perl\msrc\davint>perl Makefile.PL
     Will use Apache in c:\programme\apache group\apache
     Will use mod_dav in C:\perl\msrc\cvs\mod_dav
     Checking if your kit is complete...
     Looks good
     Writing Makefile for Apache::Array
     Writing Makefile for Apache::DAV::Datum
     Writing Makefile for Apache::DAV::DynContext
     Writing Makefile for Apache::DAV::DynHooks
     Writing Makefile for Apache::DAV::DynModule
     Writing Makefile for Apache::DAV::DynProvider
     Writing Makefile for Apache::DAV::Error
     Writing Makefile for Apache::DAV::HooksDb
     Writing Makefile for Apache::DAV::HooksLiveprop
     Writing Makefile for Apache::DAV::HooksLocks
     Writing Makefile for Apache::DAV::HooksRepository
     Writing Makefile for Apache::DAV::IfHeader
     Writing Makefile for Apache::DAV::Lock
     Writing Makefile for Apache::DAV::LockDB
     Writing Makefile for Apache::DAV::LockTokenList
     Writing Makefile for Apache::DAV::LockupResult
     Writing Makefile for Apache::DAV::PropCtx
     Writing Makefile for Apache::DAV::PropsResult
     Writing Makefile for Apache::DAV::Resource
     Writing Makefile for Apache::DAV::Response
     Writing Makefile for Apache::DAV::StateList
     Writing Makefile for Apache::DAV::Text
     Writing Makefile for Apache::DAV::TextHeader
     Writing Makefile for Apache::DAV::WalkerCtx
     Writing Makefile for Apache::DAV::XMLAttr
     Writing Makefile for Apache::DAV::XMLDoc
     Writing Makefile for Apache::DAV::XMLElem
     Writing Makefile for Apache::DAV
     Writing Makefile for Apache::TableEntry
     Writing Makefile for Apache
     Writing Makefile for WrapXS
     Writing Makefile for Apache::DAV

and now compile...

How does it go on...

Generating documentation

XSBuilder already extracts source comments for functions and structures. It also parses doxygen comments, which are used in Apache 2.0. Lyle Brooks has started on automaticly createing POD files from this information.

Improving callbacks

Callbacks are the main area that needs improvement.

Bring it back to mod_perl 2.0

First version will be released just after the conference to CPAN

Any feedback and help appreciated

Questions?