Lately I’m becoming more and more aware of the possibilities that are out there for developers, both in building, testing and running your applications in the cloud.
So I changed a hobby project i’m working on to be build, tested and deployed in the cloud. and I like to share my experiences here.
Currently i’m using 3 parties to get the picture complete
- Github - Online Git repositories with nice collaboration features, if you know it, you’ll be using it, if you don’t, have a look, you’ll never look back
- Cloudbees – Free PaaS with Git Repository and Jenkins CI instance, you could also run your apps here, but no support for Scala, Lift at the time I looked at it
- Cloudfoundry – Free for now (Beta) PaaS from VMWare – Support Scala and Lift, which makes it easy for me.
Registering a app at cloudfoundry
Once you have an account and installed the command-line tools you can claim your little application space at the hosted cloudfoundry (or download and run a micro.cloudfoundry instance in your local Vmware system)
$ vmc target api.cloudfoundry.com $ vmc login [email] $ vmc push my-app --path target/war --url my-app.cloudfoundry.com --instances 2 --mem 512k --runtime java $ vmc create-service mysql my-app-db my-app
What this means is, i’m pushing my war based maven build webapplication to the hosted cloudfoundry as two instances with each 512k memory and runtime java, I’m also binding it to a mysql service called my-app-db.
To see the result of my actions I can type vmc apps
$ vmc apps +-------------+----+---------+---------------------------+-------------+ | Application | # | Health | URLS | Services | +-------------+----+---------+---------------------------+-------------+ | my-app | 1 | RUNNING | my-app.cloudfoundry.com | my-app-db | +-------------+----+---------+---------------------------+-------------+
The commandline tools also give me options to check logs and crashes, start and stop the application etc.
Database connectivity
As cloud applications/instances/services are usually not as easily identifiable with an ip-address, your application needs to find out about services (like a database) in a different manner.
I’m doing this by using the org.cloudfoundry.runtime library to create a mysql Service for me
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | object CloudFoundryConnection extends ConnectionManager { def newConnection(name: ConnectionIdentifier): Box[Connection] = { try { import org.cloudfoundry.runtime.env._ import org.cloudfoundry.runtime.service.relational._ Full(new MysqlServiceCreator(new CloudEnvironment()) .createSingletonService().service.getConnection()) } catch { case e : Exception => Empty } } def releaseConnection(conn: Connection) {conn.close} } |
This object will return a connection if it is running in the cloud, otherwise I’ll fall back to the normal database connection of in my case an H2 database which I use for development
so I also changed the DB initialisation part of Lift in its Boot class.
The order of DB discovery is now 1) jndi connection, 2) cloudfoundry 3) connection in Properties file 4) H2 database
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | if (!DB.jndiJdbcConnAvailable_?) { val connection = CloudFoundryConnection connection.newConnection(DefaultConnectionIdentifier) match { case (Full(connection) => { DB.defineConnectionManager(DefaultConnectionIdentifier, connection) } case Empty => { val vendor = new StandardDBVendor( Props.get("db.class") openOr "org.h2.Driver", Props.get("db.url") openOr "jdbc:h2:lift_proto.db;AUTO_SERVER=TRUE", Props.get("db.user"), Props.get("db.password") ) LiftRules.unloadHooks.append(vendor.closeAllConnections_! _) DB.defineConnectionManager(DefaultConnectionIdentifier, vendor) } } } |
Maven integration
Cloudfoundry also provides a maven plugin that allows most of the functionality to be run as part of a maven goal, which is ideal when you want to deploy from your Jenkins build.
Here are the changes to my pom.xml
Add the repository for the org.cloudfoundry.runtime dependency
<repository> <id>springsource-milestones</id> <name>SpringSource Milestones Proxy</name> <url>https://oss.sonatype.org/content/repositories/springsource-milestones</url> </repository>
Add the dependency itself
<dependency> <groupId>org.cloudfoundry</groupId> <artifactId>cloudfoundry-runtime</artifactId> <version>0.6.1</version> </dependency>
For the plugin add the plugin repository
<pluginRepository> <id>repository.springframework.maven.milestone</id> <name>Spring Framework Maven Milestone Repository</name> <url>http://maven.springframework.org/milestone</url> </pluginRepository>
and the plugin itself
<plugin> <groupId>org.cloudfoundry</groupId> <artifactId>cf-maven-plugin</artifactId> <version>1.0.0.M1</version> <configuration> <server>mycloudfoundry-instance</server> <target>http://api.cloudfoundry.com</target> <url>medicate.cloudfoundry.com</url> <memory>512</memory> <instances>2</instances> </configuration> </plugin>
Username and password information you can specify in the servers section of your .m2/settings.xml file
<server> <id>mycloudfoundry-instance</id> <username>email@address.com</username> <password>s3cr3t</password> </server>
now you can update your deployed application by a simple
$ mvn cf:update
Continous Integration on Cloudbees
First setup an account at cloudbees.com, create a repository and a Jenkins instance, and make sure cloudbees has your public key to allow git pushes.
Then setup a maven 2/3 build in Jenkins, setting the scm section to watch for changes from your cloudbees git repository.
Make sure your build name doesn’t have any spaces in it, cloudbees will choke on that, at least with the scala compiler.
As we need the login information from the .m2/settings.xml, you can upload this through webdav to repository-[username].forge.cloudbees.com/private directory.
And then in our jenkins build set /private/[username]/settings.xml in the alternative maven settings field (advanced button in the maven section)
I’ve set the maven goal to clean scala:doc cf:update meaning it will compile, test, create scala docs and deploy the application to cloudfoundry.
to finish it up you need to add a remote to cloudbees in your local git repository and push your changes. this then will trigger a build and deploy your app.
git remote add cloudbees ssh://git@git.cloudbees.com/[username]/my-app.git
To conclude, besides the 2 i’ve mentioned, there are a number of solutions out there, that all work in similar ways, but each still has it own requirements, If you want to run Java, Ruby, python, php, node.js applications, with Mysql, postgres, MongoDb, CouchDb, or RabbitMQ, you can get it to work on one or more of these providers.
Last but not least here is a picture from the actual application i’m building to visualize all I’ve talked about above:

In the company I work in we have some complex tags that are not easily unit-testable because there are some dependencies between them, the best option here should be to re-factor them, unfortunately as this would break backward compatibility, we have a process of deprecation to go through until we can actually do that.
But code with hardly any tests on them always makes me feel uncomfortable. so I decided to do some (almost integration) tests but running as part of the normal unit test suite
I’m doing this by placing the tags on a jsp, running this on a embedded Jetty Server and using Selenium Webdriver to test the resulting html.
First the base class, which takes care of starting/stopping the embedded Jetty Server and setting up the web driver.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | public abstract class BaseTest { private static int port = 9595; private static Server server = new Server(port); protected static String baseUrl = "http://localhost:" + port + "/"; protected static WebDriver driver; @BeforeClass public static void setup() throws Exception { //configure jetty as an exploded war URL webAppUrl = BaseTest.class.getClassLoader().getResource("/"); WebAppContext wac = new WebAppContext(); wac.setContextPath("/"); wac.setWar(webAppUrl.toExternalForm()); server.setHandler(wac); server.setStopAtShutdown(true); //makes sure the sever is stopped even if the @AfterClass is never reached server.start(); //HtmlUnit drive doesn't give a popup, the rest of the drivers do driver = new HtmlUnitDriver(); } @AfterClass public static void teardown() throws Exception { driver.close(); server.stop(); } |
In my project I have setup a resources directory that contains a WEB-INF/ with a web.xml containing my custom tag definitions and my tld file as you would with a normal web project
Webdriver comes with a set of Drivers for different browsers (Firefox, chrome, IE, iPhone, Android, etc) for testing browser compatibility, I don’t care about that, the tags I’m testing do not output any html/css that might require this.
The HtmlUnit driver is internal and doesn’t popup a browser window so ideal for my purpose.
Writing a test now becomes very easy:
In this case I have a custom tag that when invoked simply outputs “Hello World”
my test jsp:
1 2 3 4 5 6 7 8 9 10 | <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" isELIgnored="false" %>
<%@ taglib uri="/customtaglib" prefix="custom" %>
<html>
<head>
<title>Hello World</title>
</head>
<body>
<p><custom:helloworld /></p>
</body>
</html> |
The Test class:
1 2 3 4 5 6 7 8 9 10 | public class HelloWorldTest extends BaseTest { @Test public void testHelloWorldTag() { driver.get(baseUrl + "helloworld.jsp"); Assert.assertEquals("Hello World", driver.getTitle()); WebElement element = driver.findElement(By.cssSelector("body p")); Assert.assertEquals("Hello World", element.getText()); } |
Testing the resulting html is made very easy, by having the WebElement findElement() and List
and then setting predicates with the By class. for instance By.cssSelector(), By.tagName() etc.
It is also possible to test click throughs and form submits, by calling click() or submit() on WebElements.
Added bonus is that although the tag code is running in the embedded jetty server and not in the unit tests itself it’s coverage is measured by our code coverage tools
One thing to look out for is for the embedded Jetty to work with JSP’s you need to add jetty’s version of the jsp spec to your project not the javax.servlet.* ones.
my maven dependencies look likes this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | <dependencies> <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>2.12.0</version> </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> <version>6.1.22</version> </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jsp-api-2.1</artifactId> <version>6.1.14</version> </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jsp-2.1</artifactId> <version>6.1.14</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.7</version> <scope>test</scope> </dependency> </dependencies> |
In this blog post I would like to do a basic example on how to do map reduce using actors
According to Wikipedia:
In computer science, the Actor model is a mathematical model of concurrent computation that treats “actors” as the universal primitives of concurrent digital computation: in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.
The Actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-oriented programming languages, but differs in that object-oriented software is typically executed sequentially, while the Actor model is inherently concurrent.
An actor is a computational entity that, in response to a message it receives, can concurrently:
- send a finite number of messages to other actors;
- create a finite number of new actors;
- designate the behavior to be used for the next message it receives.
There is no assumed sequence to the above actions and they could be carried out in parallel.
A number of languages have the actor model implemented in their core (Io, Erlang, Scala), for most other languages it’s available as a library (Akka for Java, Retlang for .Net, Haskell-Actor for Haskell, Parely and Pykka for Python)
In this example I’ll be using the Akka framework for Java and Scala, which brings actors to the enterprise level by adding for instance fault tolerance (supervisors, let it crash semantics) and remote actors, If you want to play around with this yourself the best place to start is to use the typesafe stack which includes Akka and Scala
I’m using a typical map/reduce use case, namely counting words in a document
I do this by having a ‘master’ actor that divides the work in lines, have a number of ‘worker’ actors that count the words for a single line, replying the result to the ‘master’ to aggregate
Messages that actors send to each other are simply objects (case classes, the receive method typically uses pattern matching to determine which message has arrived), multiple messages to an actor will be queued, and handled sequentially
So let’s create the message objects
30 31 32 33 34 | // Messages sealed trait MapReduceMessage case class CountDocument(document: Iterator[String]) extends MapReduceMessage case class CountLine(line: String) extends MapReduceMessage case class Result(values: Map[String, Int]) extends MapReduceMessage |
The CountDocument which will be send to the master, the CountLine to each worker, and each worker will return a Result when done
as each receive method returns a Partial Function[T] which allows for several classes to handle all the messages, the sealed trait super type will actually generate a warning when not all messages are handled. neat trick that.
It will also make it impossible to add messages outside of your class
which could be something you want. or not.
Now on to the Master
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | // Master Actor, creates Worker Actors, distributes work and gathers results class Master(nrOfWorkers: Int, latch:CountDownLatch) extends Actor { val workers = Vector.fill(nrOfWorkers)(actorOf[CountLineWorker].start()); val router = Routing.loadBalancerActor(CyclicIterator(workers)).start(); val resultMap = new HashMap[String, Int](); var start : Long = _ var count : Long = 0 def receive = { case CountDocument(lines : Iterator[String]) => lines.foreach(line => if (!line.isEmpty) { count = count+1; router ! CountLine(line) }) //shutdown actors router ! Broadcast(PoisonPill) router ! PoisonPill case Result(values: Map[String, Int]) => for ((key, value) <- values) { resultMap.put(key, resultMap.getOrElse(key, 0)+value) } count = count - 1; if (count <= 0) self.stop() } override def preStart() { start = System.currentTimeMillis } override def postStop() { val end = System.currentTimeMillis-start println("Result after %s ms :".format(end)) for((key, value) <- resultMap.toList.sortBy(_._2).reverse) { println("%s: %s".format(value, key)) } latch.countDown() } } |
An actor class needs to implement the Actor interface which has one method called receive
The Routing.LoadbalancingActor is a built in actor that forwards messages send to it according to the scheme passed to it, in our case in a round robin style.
When receiving a CountDocument messages it will send each line to a worker, it will then tell the workers and the router to shutdown by sending them a PosionPill message.
When receiving a Result message it will aggregate the result and after receiving all results it will stop itself.
we override the preStart() and postStop() methods to allow us to calculate spend time and reporting the result
the ! in
router ! CountLine(line)
is a so called bang method, meaning fire and forget, there are also methods that will reply eventually (!!) and reply eventually with a Future (!!!)
The Worker looks as follows:
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | //Actor that counts the words for a single line class CountLineWorker extends Actor { def receive = { case CountLine(line) => self reply Result(countWords(line)) } def countWords(line: String):Map[String, Int] = { val result = new HashMap[String, Int] "[^A-Za-z0-9\u0020]".r.replaceAllIn(line, "") .split(" ") .foreach(word => { result.put(word, result.getOrElse(word, 0)+1) }) result } } |
Should speak for itself, sanitizes and then count the words in a line, and replies to the sending actor with the result.
now wrap it in a main class that loads the document and sends it off to the master
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | object MapReduce extends App { countWordsInFile("src/main/resources/test.txt", 6); def countWordsInFile(fileName: String, nrOfWorkers: Int) { val source = scala.io.Source.fromFile(fileName) val document = source.getLines() val latch = new CountDownLatch(1) val master = actorOf(new Master(nrOfWorkers, latch)).start() master ! CountDocument(document) latch.await() source.close() } } |
This will load the document and sends it off to the master also instructing it to create 6 workers, as actors are called asynchronously, the Countdown latch is there to prevent the program from stopping before all the work is done.
The full source code is available as a gist on github
A couple of other interesting Akka features are
Become:
you can tell each actor to become another, this allows for hot code swapping, which is essential in fault-tolerant, always up systems.
def receive = { case updateYourSelf(newMe:ActorRef) => { become newMe } }
Typed Actors (example blatantly stolen from Akka documentation):
extending any POJO with TypedActor will turn them into actors
your class need to have a separate interface/implementation for this to work
trait RegistrationService { def register(user: User, cred: Credentials): Unit def getUserFor(username: String): User } public class RegistrationServiceImpl extends TypedActor with RegistrationService { def register(user: User, cred: Credentials): Unit = { ... // register user } def getUserFor(username: String): User = { ... // fetch user by username user } } val service = TypedActor.newInstance(classOf[RegistrationService], classOf[RegistrationServiceImpl], 1000) //last parameter is timeout for Futures
Remote Actors (example blatantly stolen from the Akka homepage):
// server code class HelloWorldActor extends Actor { def receive = { case msg => self reply (msg + " World") } } remote.start("localhost", 9999).register( "hello-service", actorOf[HelloWorldActor]) // client code val actor = remote.actorFor( "hello-service", "localhost", 9999) val result = actor !! "Hello"
As our website grows and people can log in, add books to their own collection, a need to access this collection on their mobile arises.
When starting the mobile application and entered the correct login information, a request is being made to get the list of authors.
Clicking on an author will get a list of books from that author that the user owns, clicking on a book shows the details.
So the first query that needs implementing is: Give me a list of Authors for which the current user owned books
Given the Following Mapper Data Model:
Author <- (1) -> Book <- (2) -> User
(1) An Author can have more books while books can be written by multiple authors (N-N)
(2) A user can own multiple books, while a book can have multiple owners (N-N)
note: When successfully authenticating in a user his API key is returned which is used in all other requests
Below we will see the code that executes this query
what happens line by line is this:
- Line4: we get a list of books for the current user
- Line5: we map the colllection to one that contains the authors for each book, the result is a list of lists of authors List[List[Author]]
- Line6: foldLeft applies the method to each element in the list, from left to right, as the method ++ means adding two list’s together, we’ll end up with one list containing all authors
- Line7: distinct removes any duplicates in there
- Line8: We sort the list on the author’s lastName property
- Line9: we map it to a List of Xml fragments
Scala’s inline XML support allows us to simple put the Authors tag around it.
The second case does the same but gives a JSON response
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | serve { case "api" :: key :: "user" :: "authors" :: "all" :: _ XmlGet _ => { <Authors> { Book.findAll(In(Book.id ,BookUser.book, By(BookUser.user, getUserIdFromKey(key)))) .map(book => { book.authors.get }) .foldLeft(List[Author]())(_ ++ _) .distinct .sortWith((a1:Author, a2:Author) => a1.lastName.is < a2.lastName.is) .map(author => author.toXml) } </Authors> } case "api" :: key :: "user" :: "authors" :: "all" :: _ JsonGet _ => { JsonWrapper ("Authors", { Book.findAll(In(Book.id ,BookUser.book, By(BookUser.user, getUserIdFromKey(key)))) .map(book => { book.authors.get }) .foldLeft(List[Author]())(_ ++ _) .distinct .sortWith((a1:Author, a2:Author) => a1.lastName.is < a2.lastName.is) .map(author => author.toJson) }) } } //helper method def getUserIdFromKey(key:String):Long = { User.find(By(User.key, key) match { case Full(user) => user.id case (_) => 0 } } |
As the web application I’m currently building will have a mobile app, a REST Interface is a nice clean way for the mobile app to connect, retrieve and update information from the website.
fortunately Liftweb makes it very easy to do this, all you need to do is extend your class with the RestHelper Trait and let liftweb know about it in your Boot.scala
Lets say we want to get a list of all books in the system when we access http://(server)/api/books/all
First we need to tell Lift we have a class that responds to requests in Boot.scala:
LiftRules.dispatch.append(BookRestApi)
then the class itself: BookRestApi.scala:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | object BookRestApi extends RestHelper { serve { // list of books case "api" :: "books" :: "all" :: _ XmlGet _=> { <Books> { Book.findAll.map(book => book.toXml) } </Books> } case "api" :: "books" :: "all" :: _ JsonGet _=> { JsonWrapper("books", Book.findAll.map(book => book.toJS)) } } } |
The RestHelper trait parses the Url and makes the relevant parts available for Scala’s pattern matching capabilities.
for the first case statement:
- the api :: books : all List will match if the url points to /api/books/all
- the XmlGet method will match if the client has indicated he can accept xml by setting an accept header or by ending the url with a .xml suffix
Scala’s inline XML capabilities make it very easy, to envelop the result in a <Books> element.
the second statement does the same but then when the client requests a json response.
For the Json part I used a JsonWrapper helper method because the inline code looked totally unreadable, this still isn’t very readable but at least it has a descriptive method name now
1 2 3 | def JsonWrapper(name : String, content : JValue) : JValue = { (name -> content) } |
If you would call this method with the name “Books” and the content {“foo” : “bar” } it will output :
{"books" : {"foo" : "bar"} }
This shows both a positive and negative aspect to the Scala language, it makes for a very flexible and fast development, but loses a lot in readability, which makes maintenance more difficult.
Now if would want to get a single book when I specify the id of that book, I’d add the following:
1 2 3 4 5 6 7 | case "api" :: "books" :: AsLong(id) :: _ XmlGet _=> { Book.findByKey(id) match { case Full(book) => {book.toXml} case (_) => NotFoundResponse("Book with id " + id + " not found\r\n") } } |
This pattern will only match on /api/books/{id} and only if that Id can be converted to a Long, in which case it will try to look up a book with that id
if the book isn’t found (the Box returned from findByKey isn’t Full) it will give the text back
In the next article I’ll talk about PUT requests and authentication so we can add books through the rest interface
In this second article I want to focus on the persistence layer of Lift by taking you through the following use case:
As an admin I want to be able to add/edit news articles and choose whether or not they appear on the homepage
For this we need to store the articles in a database, A template and Snippet to show them and some way of managing them.
Lift has 2 built-in persistence frameworks, Mapper and Record, but also JPA is fully supported.
As Mapper has CRUD and User management I’ve chosen to go with that one.
Our News article will have a Title, an Author, the content and a boolean that allows it to be seen on the homepage
1 2 3 4 5 6 7 8 9 | class News extends LongKeyedMapper[News] { def getSingleton = News object title extends MappedString(this, 100) object author extends MappedString(this, 100) object content extends MappedTextarea(this, 4000) object showOnHomepage extends MappedBoolean(this) } object News extends News with LongKeyedMetaMapper[News] |
* A class and a object with the same name are called companion’s in Scala and the relationship is that the object is a singleton instance of the class, and they have access to each others private methods.
by extending News with LongKeyedMapper[News] we are able to persist it to the database.
Now we will mix-in some traits to add some extra functionality, a Trait in Scala is like a Java interface but allows for partial implementation. Say we want to add a primary key field and two date fields to store created and updated values, all we need to do is add the triats: IdPk (for primary key) and CreatedUpdated to the class definition.
1 | class News extends LongKeyedMapper[News] with IdPK with CreatedUpdated |
this will add the MappedLong object id, and 2 MappedDate objects created and updated to the News class
To add CRUD functionality we just add the CRUDify trait, this one needs to be added to the object instead of the class so that one now looks like:
object News extends News with LongKeyedMetaMapper[News] with CRUDify[Long, News]
This will add the forms, validation and handling to the news object.
To hook this all up into Lift’s bootstrapping we have to add some lines to Boot.scala, which is the class that lift calls before anything else.
The following line will make sure Lift knows about our persistence class and creates the database tables if needed
Schemifier.schemify(true, Schemifier.infoF _, News)
Lift works with a Sitemap, any page you want Lift to serve needs to be added to the sitemap. so we need to add our CRUD pages there fortunately CRUDify has a convenient helper method for this.
val menu = Menu(Loc("HomePage", "index" :: Nil, "Home Page", Hidden)) ::: News.menus LiftRules.setSiteMap(SiteMap(menu:_*))
This will create a Homepage menu item and adds the news crud pages to it.
If we want to create a news item through code we could just execute the following line:
News.create.title("A news Item").author("gertjan").content("Breaking news, an item has just been created").showOnHomePage(true).saveMe
Next the snippet that will display a number of news items on the homepage
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | class Homepage { def news(in: NodeSeq) : NodeSeq = { val articles = News.findAll(By(News.showOnHomepage, true), OrderBy(News.createdAt, Descending), MaxRows[News](5)) def bindArticles(template: NodeSeq): NodeSeq = { articles.flatMap { case (news) => bind("article", template, "title" -> news.title, "author" -> news.author, "content" -> StringHelper.unescapeXml(news.content), "created" -> news.createdAt, "updated" -> news.updatedAt) } } Helpers.bind("news", in, "articles" -> bindArticles _) } } |
The first line will lookup 5 news items that have the showOnHomepage boolean set to true and order them on date created.
The method bindArticles method will create bindings for each item found. as Lift escapes all xml/html by default, we need to unescape it especially when we want rich content.
The last line, will bind the collection to <news:articles/>
Then all is left is the html template:
<lift:surround with="default" at="content"> <div class="news"> <lift:homepage.news> <news:articles> <h3><article:title /></h3> <h4><i>by:</i><article:author /> <i>created:</i><article:created /> <i>updated:</i><article:updated/></h4> <p><article:content /></p> </news:articles> </lift:homepage.news> </div> </lift:surround>
Because the <news:articles> contains a collection, lift will loop over it and process the inner html with each item in the collection.
A thing about authorization, we want the CRUD pages (and probably some other pages) to be accessible only to logged in users, this can be handled in several ways.
1) The Menu class has a parameter that defines the visibility of it, this can be easily used to check if a user is logged in:
val adminLoc = Loc("AdminPage", "admin" :: Nil, "Admin Page", Test((u) => u = User.loggedin_?))
or override the specific menu methods that are defined in the CRUDify trait
override def editMenuLocParams = If(User.loggedIn_? _, RedirectResponse("/user_mgt/login")) :: super.editMenuLocParams
And there it is, with surprisingly little code, we now have the beginnings of a content management system.
I’ve been playing around with Lift for a while now, and I’d like to share some of my experiences with you all, although i’m inexperienced with both Scala and lift, I’m able to get things done pretty quickly in comparison with doing it in Java. which says a lot about both the language and the framework.
I’m building a webapplication that holds my collection of books, so the first use case I want to present to you and go through the implementation is the following:
As a user I want to be able to quickly import a list of isbn numbers
Lift moves away from your typical model view controller framework, but focuses on keeping your template’s designer friendly without any logic or code in there. Besides that the main focus areas are security and scalability, read the documentation on the lift website if you want to know more about this.
Taking the use case, we need to build a form with a textarea that allows us to copy/paste a list of isbn numbers in and add those to the database. Later use cases will then look up title/author information from for instance amazon, google books, or isbndb
Our html template which contains the form looks like this:
<lift:surround with="default" at="content"> <h3>Import Isbnn rumbers</h3> <lift:ImportIsbn.bulk form="POST"> <entry:text /><br/> <entry:submit /> </lift:ImportIsbn.bulk> </lift:surround>
- The <lift:surround> tells lift to ‘surround’ this template with the default.html template which contains the header, footer and navigation for our website, inserting this template at the <lift:content /> tag in that template.
- The <lift:ImportIsbn.bulk> tells Lift to look for a class named ImportIsbn with a method bulk with the inner html as a parameter for us to manipulate
- The <entry:*>’s we will bind to form elements in the snippet code later on
Something more about templates:
- if a snippet takes long to complete, surround it with <lift:lazy-load> this will show an ajax loading gif while the snippet is executed.
- if you have a lot of snippets that take a while, surround them with <lift:parallel> to execute them in different threads
now to create our form:
1 2 3 4 5 6 7 8 | class ImportIsbn var isbnnrs = "" def bulk(in : NodeSeq) : NodeSeq = { bind("entry", in, "text" -> SHtml.textarea(isbnnrs, isbnnrs = _), "submit" -> (SHtml.hidden(processIsbnNrs) ++ <input type="submit" value="Add" />)) } } |
The bind method is a helper method that looks in the html (NodeSeq) passed on for tags marked with the “entry” prefix and binds them to html elements, in good functional programming the SHtml.* methods take a function as a parameter, for the textarea, we assign the value of the textarea to it, the hidden one calls the method processIsbnNrs
we could also have used
SHtml.submit("Add", processIsbnNrs)
but the above allows us to quickly make a ajax form out of it.
To do this we simple put a Shtml.ajaxForm around the bind.
SHtml.ajaxForm( bind(.... etc ) )
This will create the javascript methods to submit this form without doing a normal post
Due to Scala’s inline xml capabilities a snippet can also return xhtml directly, lift only works with xhtml (and if you use the 2.2-SNAPSHOT even html5)
def mySnippet(in: NodeSeq) : NodeSeq = { <p>This is returned from my snippet</p> }
After submitting the method processIsbnNrs is called which looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 | def processIsbnNrs() = { isbnnrs split " " map {addBook _} } def addBook(nr : String) { Book.find(By(Book.isbn, nr)) match { case (Full(book)) => alreadyInDb += " " + nr case (_) => Book.create.isbn(nr).saveMe } msg = "Skipped: " + alreadyInDb S.warning(Text(msg)) } |
As the isbnnrs var is bound to the textarea it will now contains the list of isbnnrs, the first line will split it on a space and then calling the addBook method for each entry
The addBook method will try to find the book in the Database, (the following post will look at Mapper the persistence layer in Lift). if it doesn’t it will add it to the database.
and that is all to it, I left out validation and initializing of the var’s here. but the code is public at github if you want to take closer look.
A close observer will see the use of the method Full() in there, Scala has the Option class with which can be an instance of Some or None, which is designed to make continuous null pointer exception checking unnecessary.
Lift takes this one step further by having a Box class, which can be Full, Empty or Failure. This in combination with Scala’s match/case functionality is very powerful and creates very readable and thus maintable code.
In our case if a Book is found (the box is full) do nothing, if not save a book with that isbn nr to the database.
See the Box article on Liftweb for more information about this.
Unittests are an integral part of building quality software and the sole responsibility of the developer.
Here I explain some of the lessons i’ve learned after the last couple of years.
- Unittests takes times to write
I’m even inclined to say that writing your tests should take at least the time as writing your functionality takes. - Start with the unit tests
Start writing your unittests first, setting up just enough of your classes to get the test compiling, this will force you write your test based on the functionality required not the buggy implementation you’ve just written. - Independent
as the name says, a unit test will test a single unit of code, use integration tests to see how parts are working together. - Small
A unit test should test a small bit of functionality, one class, or even one method. I usually write extra unittests that test entire use-cases, but those work next to the normal ones. - Test negatives as wel as positives
Your first test will probably check whether the functionality works when specifying the correct parameters , but also test whether the right results are returned or the correct exceptions are thrown when specifying the wrong ones. - Use code coverage tools to measure effectiveness
Coverage tools, like for instance Clover will tell you how much of your code is covered, and also which classes are the most risk (high complexy, low coverage) - Be careful when using mock objects
Using mock objects keeps your unittests indepent but it has the danger that you are testing your mock objects instead of your own functionality, write integration tests - Don’t change outputted results into asserts
It can be very tempting to first see what your class outputs when running a test, and then writing asserts to test on those results, but you will just be validating that bug that is still in there.
I’ve been playing around with REST (REpresentational State Transfer) for a bit and decided to write a small tutorial to show how easy it is to create a rest interface to your business objects
Advantages of rest
- Works over HTTP, known protocol with known data formats (mime types), no weird ports open in your firewall
- Every resource is uniquely identified by a URI
- Can do CRUD on those resources (they are just called POST, GET, PUT, DELETE)
- used by all the major players for cloud applications (SUN’s cloudApi, Microsoft’s Azure services platform)
- we might finally get rid of webdav
I’ll be using the Jersey Framework for this.
As i’ve been working on a backend that will allow you to manage agile development processes (Scrum, Kanban etc) I thought I use that.
In this application every object is represented by a Card for instance a Story, Task, Bug, work Item, etc.
Cards have a hierarchical structure and can each have custom properties
The class looks something like this (removed methods for clarity):
public abstract class Card { private long id; private List children = new ArrayList(); private List properties = new ArrayList() }
and then for instance the Story class with some default values set in the constructor:
public class Story extends Card { public Story() { addProperty("Title", "A new story"); addProperty("Description", "as a .. I want .. for the benefit of .."); addProperty("Storypoints", "0"); } }
To be able to return these objects through REST they need to be in a format that can be distributed over HTTP, in this case we will be using XML and JSON.
The Java XML Bindings are perfectly suited to create xml representations of our classes
@XmlRootElement(name="card") public abstract class Card { private long id; @XmlAttribute(name="type") protected String instance = this.getClass().getSimpleName(); @XmlElement(name="card") private List children = new ArrayList(); @XmlElement(name="property") private List properties = new ArrayList(); @XmlAttribute public long getId() {} }
* the Entry class is a simple class with a name and value property, I choose this approach as JAXB has an issue in representing HashMaps() in XML
this will produce the following xml
for JSON we don’t need to do anything special then include the jersey-json.jar to the project.
Now to configure and hookup the rest servlet
I added the jersey servlet to my web.xml
Rest Web Service
com.sun.jersey.spi.container.servlet.ServletContainer
1
Rest Web Service
/services/*And create the service provider class
@PerRequest @Path("/agileassistant") @Consumes({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML }) public class AgileAssistantService { ... }
This will tell Jersey that this is a provider that will respond to requests to /rest/agileassistant, accept JSON and XML and can output JSON or XML depending on the accept headers sent by the client
Now if we for instance want to return the first card (rootCard) and all its children we add this method
public static Card rootCard = Card.createCard(Root.class); @GET public Card getRoot() { return rootCard; }
(in my data model the rootCard is always present and functions as a starting point to add the rest of the cards in a hierarchical structure)
So requesting this card will effectively return the entire structure for my test data:
Doing a parameterized query is almost as easy
@GET @Path("/card/{id}") public Card getCard(@PathParam("id") long id) { return getCardById(rootCard, id); }
here the id that you specify will be injected into the parameter
giving the following output
To conclude enabling rest for your applications is pretty easy, writing this blog took longer than writing the code.
In a next post I will add POST/PUT/DELETE requests to store data in your application
Data binding in JavaFX is pretty straightforward.
You should think binding to Java Objects should be just as simple, well it is a little complicater then just doing
var someObject : bind myJavaclass.getProperty();
JavaFX needs to be aware of any changes to the bound variables to be able to update the related objects. It does this automagically for all javafx objects.
It seems JavaFX uses the Observer/Observable pattern for all its internal objects, well we can do the same to bind with JavaObjects.
To get myself familiar with JavaFX i decided to write a photoviewer/editor. This application has a java backend with in particular a Photo class which contains all the information of a certain picture, filename, path, tags, exif information etcetera
To get changes in the Java class to appear in the gui I create a PhotoAdapter javafx class that extends Observer and implemented the properties I need in my GUI.
public class PhotoAdapter extends Observer { public-read var filename: String; public-read var tags: String[]; public-read var exif: String[]; public-init var photo: Photo on replace { photo.addObserver(this); filename=photo.getFilename(); tags = photo.getTags(); exif = photo.getExif(); }; override function update(observable: Observable, arg: Object) { filename = photo.getFilename(); tags = photo.getTags(); exif = photo.getExif(); }
after that I had my existing Java Photo class extend Observable and in every method that made changes I added the methods setChanged() and NotifyObservers() to inform the Observers about the change
public class Photo extends Observable { private Map<String, String> exifInformation; .... public void setExifInformation(Map exifInfo) { this.exifInformation = exifInfo; setChanged(); notifyObservers(); } }
in my main javafx class I can then use it as follows
var photoAdapter = PhotoAdapter { photo : new Photo(startImage); }
and use to fill a ListView swing object with for instance the tags:
ListView { items: bind photoAdapter.tags; width: 200; height: 300; }
et viola, thats all that there is to it


