..but do I need it?

Well that depends:

• do you have any objects in whose alteration history you’re interested in? Think about any financial or business object.
• social network app? think automatic activity feeds..
• do you have any analysis workflow that produces a single output from multiple input data points? Provenance can retroactively create the relationships between the input and output datums.
• do you want to monitor a given object’s access patterns temporary? Think debugging.

Actually, the longer you think the more use cases appear.

After the break I’ll talk about my master thesis (that dealt with provenance), the corresponding RoR prototype and future plans w.r.t. provenance within Ruby on Rails.

Background Information

I’ve spend some time with my master thesis focusing on provenance. It’s academic so do not expect readability or real-world usability. For that there’s a short paper detailing the rails prototype, an additional use case was detailed in another paper. All source code can be found on github, it might take some days to get the newest versions into the repositories.

My basic idea was that developers do not want to deal with the details of provenance gathering or storage. They should focus upon their domain application. I developed a system that allowed automatic provenance capturing (through a rails plugin), networked storage and dedicated network query facilities.

The prototype did work, unobtrusive provenance capturing is perfectly possible in Ruby on Rails. Why do I believe that the master thesis prototype is not perfectly suited for real-world applications? Well,

• storage was done through a separated RDF graph engine, encapsulated within an own RoR application. This is well suited for large academic test applications but I believe that something using an embedded storage solution (thus easier to integrate within existing systems) might be better suited for Rails.
• provenance queries had to be done through SPARQL or a special (easy) REST-based interface. An interface directly integrated within Ruby would be preferable.
• It’s all still Rails 2.3 based. I’ve finished the prototype some time ago and did work the last months mostly on the documentation. So the prototype should be forward-ported to Rails 3.0.0beta3.

Towards a bright new future!

So I have finally finished my master thesis and would need a embeddable provenance solution for one of my side-projects. I will base it upon the work done during my master thesis but it will differentiate in various points:

• use embedded storage. Application developers should be able to install the plugin, call a rake task and enjoy the benefits of automatic provenance gathering. Does anyone know a good ruby only RDF/graph engine?
• no more special query language.. users should not have to learn a new query language nor interfaces. Provenance should be be provided through virtual ActionModel methods and act as ‘normal’ object methods.
• update the plugin to work with Rails 3.0, investigate Rack and ActionModel
• keep the plugin as simple as possible.. so other people can play with it too.

I’m not perfectly sure when my side-project will need need that functionality (or if it even gets towards that point), but if you’re interested in provenance on rails just watch the repository or, even better, branch it and get involved..

Ruby on Rails provides various helpers for generating dynamic web content but sometimes you need documents that users can easily store and share between them. The ubiquitous file format for this is Adobe PDF nowadays.

The common solution fro this problem is Prawn [github, introduction]. Alas it requires a custom DSL for document description, no existing Rails views or partials can be reused. An alternative is princeXML which transforms HTML/CSS into pdf through an external binary. This would allow reuse of existing templates and knowledge (think CSS designers) but has the downside of its price tag of $3800.

The solution

Enters wicked_pdf: it utilizes wkhtmltopdf to create a PDF document from a Rails HTML template. HTML rendering is done through the well-known webkit-engine. This allows developers to do PDFs in the right way™: define the document’s structure through a simple HTML document and theme them through CSS. You’ll get the automatic benefit of themability: exchange the CSS and you have another format. There are also lots of CSS artists out there that can supply you with different designs.

I’ve used this solution with a test Ruby on Rails 3.0-beta application, the steps involved were:

Install wkhtmltopdf:

This program will convert the HTML into the PDF format. To make it work you’ll need at least version 0.9 which isn’t installed in Ubuntu 9.10 by default. Just download the static compiled version from the wkhtmltopdf website and place it under /usr/local/bin.

Install the wicked_pdf plugin:

Just install it in the usual Rails way:

git submodule add git://github.com/mileszs/wicked_pdf.git vendor/plugins/wicked_pdf

Additionally a sample configuration file is needed. You could use the plugin’s generator script for it but alas this didn’t work for me with Rails 3.0. Let’s just copy it form the plugin’s directory into config/initializers.

$ cp vendor/plugins/wicked_pdf/generators/wicked_pdf/templates/wicked_pdf.rb config/initializers

and alter the wkhtmltopdf path within it:

  WICKED_PDF = {
       :exe_path => '/usr/local/bin/wkhtmltopdf-amd64'
  }

Add pdf instructions to your controller..

For example I’m using the invoice#index action to render a simple PDF document.

  format.pdf do
          @example_text = "some text"
          render :pdf => "file_name",
                 :template => 'offers/show.pdf.erb',
                 :layout => 'pdf',
                 :footer => {
                    :center => "Center",
                    :left => "Left",
                    :right => "Right"
                 }
  end

We are rendering the view with a predefined footer containing some sample “center”, “left” and “right” strings. The render :pdf call has various options which can be seen on the wicked_pdf homepage.

.. and create PDF templates

The other half of the PDF templates is the view code consisting of a special layout and template for pdf generation.

The layout (app/views/layouts/pdf.html.erb) resembles a normal Ruby on Rails layout file:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <%= wicked_pdf_stylesheet_link_tag "pdf" %>
  </head>
  <body>
    <div id="content">
      <%= yield %>
    </div>
  </body>
</html>

The view code should not be too surprising, just place it under app/views/offers/index.pdf.erb :

<div id="someid"><%= @example_text %></div>

Lets also create a simple CSS file (public/stylesheets/pdf.css):

#someid {
  margin-left: 430px;
  display: float;
  height: 150px;
  background-color: green;
  width: 250px;
}

This is the whole rendering code.

Reference the PDF view and download it

We still need a reference to the newly created rendernig link, we can easily create this through:

link_to 'Create PDF document', invoice_path(@invoice, :format => :pdf)

When the link is clicked a pdf document will be generated and downloaded.

Conclusion

wicked_pdf allows easy pdf generation in a very Ruby on Rails’ way. It’s free, it works and is easy to employ..

..but not everything is perfect within the wicked_pdf world, especially error handlnig is lacking sometimes. When you’ve debugging initial problems you can expect the only error feedback to be a HTTP return code of 406. Just start with limited controller (rendering) options and double check that you’ve referenced the right view and layout paths and you should be fine.

• Compared to PHP, Python or Java there are just too few communication libraries. And even the libraries that exist are lacking fundamental features and almost all of them come without useable documentation. That there is no encryption (SSL or TLS) support in the whole Ruby 1.8 mail libraries is an outright shame. The IMAP library is so clumsy that a web company sells their own (still not perfect library) through their online store and they seem to make a good buck with it.

  On the Java side is just a completly different picture: Lots of libraries, even documented ones are available. After having to touch the TMail (rail’s MIME mail handler library) API using javax.mail is a heavenly gift. And it seems that the TMail generated MIME messages were invalid in a couple of cases. Not the best feature of a support library. Also the chance of finding unsolved known bugs and errors seems to be lot smaller in Java.

• This brings me to another point: I might change my opinion on Java’s constraints on its users. Java tries hard to prevent errors (i.e. the forced exception catching). I always thought that that took too many stylish possibilities away from the user, but by now I must confess that I think that this is exactly what I want from something that I’m using on the network side. I’m a lazy programmer, I want to be reminded and forced to write secure and stable code. This is the quite different to Ruby and Rails ‘make it easy for the programmer’ attitude.
• Background processing is hard. As Ruby on Rails is not multi-thread safe you can’t just spawn a thread if you need to perform some longer running task. Another disadvantage of using a single-process model is that the long running request will occupy one rails worker (i.e. rails cluster process) until it has finished – in our case that costs us around 60MB of memory per long running request, even if it is just waiting for some simple SMTP feedback. If you can find a situation where you can delay the execution of a network related task (and if you don’t you’re not thinking) for two seconds, 6 requests per seconds will DoS a standard rails cluster.

  The only solution that’s actually usable is BackgroundRb. But projects that just change their background communication system just don’t sound to production-grade ready for me. Also the admin start/stop scripts for their background server didn’t work too well for me.

• For a language that interferes it’s object’s attribute types directly from the database the ActiveRecord layer is weak. Don’t get me wrong, I understand that the Simplicity is needed to make it easily usable but I ran into various situations where I’d love to have a full blown ORM behind me. One feature that is needed quite often by our application is inheritance. ActiveRecord only offers single table inheritance, and even there you have to make sure that each row is valid (ActiveRecord should have all needed information to do that by itself BTW) or you will run into problems later on. One problem is, that it tries to abstract too much functionality away from the database while not provided as advanced interfaces by itself. Data integrity handling? Abstracted away by rails, so all databases can be used the same. The drawback is, that the data constraints and relations are fully handled by rails and not passed on to the database. Any process that might produce invalid data (e.g. a faulty rails component) might corrupt the data. Rails is able to handle that cases by itself (due to ducktyping and very few default checks), but access that data with any other framework and it blows up directly into your face.
• Transaction handling. Just try it. Then cry. Also I’m not sure if Transaction handling is even done on database level or in Rails (as done with constraints). If the later is true, it’s acutally not worth anything as soon as more processes try to access the database.

The library and documentation problems where the main reason for me to reimplement the mailing and campaigning backend in Java. The front end is still a Rails application — which is exactly what Rails is for. As I’m no friend of blown-up EJB based solutions I’ve choosen a simple Spring and JPA based solution for that problem.

I’m currently testing the last features and replacing the Ruby code part by part. As soon as I’ve done that another blog post will examine the two implementations, how much time was spent on coding them and how they perform when compared to each other.