Java HTML Parsing Example With htmlparser

Tue, May 30, 2006 Every week, I post javablogs top 10 most read blog entries on this blog. The reason for it was that I don't look at what's happening on the week-end and this will pickup interesting stories from the weekend, and I also don't watch javablogs everyday. Overall I find it quite good to be uptodate with interesting stuff happening on javablogs.

As mentionned in an earlier post my library of choice to do the parsing is htmlparser (on sourceforge) because it's free, open source and because I am lazy and did not want to do my own. If you know a better open source library, feel free to add a comment about it, I'll be glad to hear about it. htmlparser is not the easiest library to use, there are many entry points and it's not immediately clear which one to choose. So I post here how I used it if it can save a few minutes to people having to do this task.

  private static Entry parseEntry(String content) throws ParserException
  {
    final Entry entry = new Entry();

    final NodeVisitor linkVisitor = new NodeVisitor() {
      
      @Override
      public void visitTag(Tag tag) {
        String name = tag.getTagName();

        if ("a".equalsIgnoreCase(name))
            {
              String hrefValue = tag.getAttribute("href");
              if (hrefValue != null && !hrefValue.startsWith("http://"))
              {
                if (!hrefValue.startsWith("/")) hrefValue = "/"+hrefValue;
                hrefValue = "http://javablogs.com"+hrefValue;
                //System.out.println("test, value="+hrefValue);
              }
              if (hrefValue != null)
              {
                hrefValue = hrefValue.replaceAll("&", "&amp;");
                tag.setAttribute("href", hrefValue);                
              }
            }
      }
    
    };
    
    NodeVisitor visitor = new NodeVisitor() {

      @Override
      public void visitTag(Tag tag) {        
        String name = tag.getTagName();
            if ("span".equalsIgnoreCase(name) || "div".equalsIgnoreCase(name))
            {              
              String classValue = tag.getAttribute("class");
//                LOGGER.debug("visittag name="+name+" class="+classValue+"children="+tag.getChildren().toHtml());
              if ("blogentrydetails".equals(classValue))
              {
                Pattern countPattern = Pattern.compile("Reads:\\s*([0-9]*)");
                Matcher matcher = countPattern.matcher(tag.getChildren().toHtml());
                if (matcher.find())
                {
                  String countStr = matcher.group(1);
                  entry.count = new Integer(countStr).intValue();
                }
                
              }
              else if ("blogentrysummary".equals(classValue))
              {
                try
                {
                  tag.getChildren().visitAllNodesWith(linkVisitor);
                }
                catch (ParserException pe)
                {
                  LOGGER.error(pe,pe);
                }
                entry.description = tag.getChildren().toHtml();                 
                entry.description = entry.description.replaceAll("\\s+", " ");
              }
              else if ("blogentrytitle".equals(classValue))
              {
                try
                {
                  tag.getChildren().visitAllNodesWith(linkVisitor);
                }
                catch (ParserException pe)
                {
                  LOGGER.error(pe,pe);
                }
                entry.title =tag.getChildren().toHtml(); 
                entry.title = entry.title.replaceAll("\\s+", " ");
              }              
            }
            
      }

    };
    Parser parser = new Parser(new Lexer(new Page(content,"UTF-8")));
    parser.visitAllNodesWith(visitor);
        if (entry.title != null)
        {
          return entry;
        }
        else return null;
  }

First Steps With EhCache

Thu, May 4, 2006

If you need to cache objects in your system, Ehcache is a simple cache written in Java, widely used and well tested. I will present here a short tutorial on how to use EhCache for people who don't want to look around the documentation at first, but just want to test if it works in their project and to see how easy it is to setup.

Installation

Download Ehcache from the Download link on http://ehcache.sourceforge.net. Current release is 1.2.

Unpack Ehcache with an unpacker that knows the tgz format. For unix users, it is trivial, for windows users, 7zip is a free (and open-source) unpacker. It is probably the most popular, but there are other ones like tugzip or izarc or winrar.

In your java project you need to have ehcache-1.2.jar, commons-collections-2.1.1.jar and commons-logging-1.0.4.jar (versions numbers may vary) in your classpath, those libraries are shipped with ehcache.

Cache Configuration

Write an ehcache.xml file where you describe what cache you want to use. There can be several files per project, several cache descriptions per file. I use here a persistent cache. Configuration file is well described at http://ehcache.sourceforge.net/documentation/configuration.html

</ehcache>

Code

static  {  
 //Create a CacheManager  using a specific  config file  
 cacheManager = CacheManager.create(TestClass.class.getResource( "/config/ehcache.xml"));
 cache = cacheManager.getCache("firstcache");
} 

/**
 * retrieves value  from cache if exists
 * if not  create it and add  it to cache */
public String doit(String key, String value) {
 //get an element  from cache by key
  Element e = cache.get(key);
 if (e != null)    {
   value  = (String)e.getValue();
   LOGGER.info("retrieved "+ value+" from cache ");
 }
  else   {
   value = "new value" ;
   cache.put(new Element(key, value));
 }
  return value;
}

/**
 * refresh value  for given key */
public void refresh(String key) { cache.remove(key); }

/**
 * to call eventually  when your application is  exiting  */
 public void shutdown() { cacheManager.shutdown(); }

Conclusion

Using EhCache is as simple a using a Java Map with an additional configuration file.

Null vs. Errors

Wed, Apr 26, 2006

I am not particularly a fan of JCS (Jakarta Cache System) as I find ehcache code very clean and simple. But I have to say the author has some good comments on the site:

Nulls vs. Errors

I started to support ObjectNotFoundExceptions for failed gets but the overhead and cumbersome coding needed to surround a simple get method is ridiculous. Instead the JCS return null.

For having seen too many times the ObjectNotFoundException "pattern", I can only agree!

Algorithms in Java (Third Edition) Book Review

Wed, Apr 26, 2006 The book Algorithms in Java is huge, but unlike the usual huge books, the content is very interesting. It can be used as reference material, or as toilet book (to learn things while you're wasting time in the toilets).

You will learn simple things, like what is the "raison d'être" of linked lists. The author gives very good examples to illustrate his propositions. He explains through the sieve of erathostene and through Josephus problem the advantages of arrays or linked list.

You will learn step by step everything that is to be known in algorithms. Recursion, divide and conquer, Tree knowledge will be useful for the later sorting and searching chapters.

The chapter on Hashing will make you understand very clearly why the source of String.hashCode() is

public int hashCode() {

  int h = hash;

  if (h == 0) {

     int off = offset;

     char val[] = value;

     int len = count;

     for (int i = 0; i <  len; i++) h = 31*h + val[off ++];

     hash = h;

  return h;

There might be too much info on different types of sorting algorithms and the book becomes there more a reference book than anything else. But overall, you will learn plenty with this book. It is very well written, complete, and will refresh one's memory. I find it useful to read back things I learnt after a few years as I then have a very different view of the subject, and I pay closer attention to some details I completely missed the first time (sometimes).
Tags: java book review

Caching HTTP Responses in Java

Wed, Apr 19, 2006

Caching HTTP response can dramatically improve performance of your app if what you generate is in reality not very dynamic. There are many free caching frameworks in Java. Most popular seem to be ehcache, oscache, jcs and JBoss Cache.

ehcache is quite simple to use and its code is clean. They have a CachingFilter that you can put in your webapp server to cache transparently HTTP responses. However as the framework only allows you to store Objects (which makes sense for most uses), I was wondering how they cached the HttpResponse which is a stream. I was a bit disappointed by the answer, they just create a copy ByteArrayOutputStream and call toBytes() to store it in the cache. While this is optimal for a memory cache store (the whole response will anyway be in the cache, although I am not sure if they check for particularly big responses to avoid caching those or try to cache those) I don’t think it is that good for a disk cache store.

Ideally one would like the response to be stored using a buffer, to avoid having the whole response in memory. This would enable a much higher concurrent use. I think it is doable by writing your own CachingFilter and by using the concurrent utils Queue to block writing when the buffer is full.

I googled for this kind of stuff without success. I only found solutions similar to ehcache one (for example sun CachingResponseWrapper and CachingFilter or oscache CacheFilter (a bit more careful, but still a toBytes())) I wonder why it is not already done and public.

Javablogs.com 2005 Top 20

Tue, Apr 18, 2006 Most read in 2005

New in Hibernate 3: Criteria API enhancements (816): Projection, aggregation, subselects, detatched criterias - its all there in the Hibernate 3 Criteria API. Let me show you some examples, starting with the new projection API.… [read]

Hello, IDEA! (555): From the recent Java IDE discussions, it seems like there's a good portion of Java programmers who don't know IntelliJ IDEA, or simply haven't tried it yet. In this short screencast,… [read]

Hey Gosling: This is why we don’t use Java 5.0 yet! (461): Whenever there is a major JDK update everyone on the Sun Microsystems side seems to love to beat the drum of upgrading.… [read]

MSN7.5?? (450): ?????Google Talk????????????????????????????????????Google Talk???????? Google Talk???????????????????????????????????? [read]

The worst code I've ever seen. Yes, that's true. (422): I've been in this business for 25 years and have been programming since the mid seventies. Even when I was a young pup, full of bright ideas and hubris and those around me were at least as bad,… [read]

JSP is officially dead (407): Well, it looks like with the Final Draft of Java EE 5, the final nail has been placed in JSP's coffin.… [read]

My wife is hot and she can code, too (400): My wife Keri loves puzzle games--tetris, scrabble, crosswords, text twist--any game where you have to figure something out, she's on it. She has a degree in CS, and is employed as a UI specialist,… [read]

IBM announced SOMA - Service-Oriented Modelling and Architecture (396): [read]

G-mail runs on Tomcat???!!!!! (390): Well, hello guys and gals...haven't been in the Java blog scene recently, but am glad to know Java is going strong as ever (not that I ever doubted it,… [read]

Web Services articles (374): [read]

Bill Gates tries Firefox (362): Tim Weber of the BBC reporter rustles up Bill Gates quote of 2005: Bill Gates is one of the people with Firefox on his computer, so I asked him for his opinion. I played around with it a bit,… [read]

Language Oriented Programming: Everything is a Language (361): Some people don't 'get' Language Oriented Programming. It's a different perspective. Once you make the mental shift, everything starts to fall into place. Over the past few months,… [read]

The killer app for Web 2.0 has arrived (356): Time to throw in the towel, 37 Signals. The future has arrived: iClock (via flocksucks) [read]

New Search Engine Blows Google Away (356): After months of incredibly secret development, PreviewSeek Limited has launched the PreviewSeek search engine. My initial impression? It blows google away with its far more powerful searches. [read]

Death to Apache (356): So our Apache heros have now decided that it isn't quite enough to prove to the world that they are abysmal failures at producing a J2EE container,… [read]

WebSphere 6.0 System Management Enhancements (356): [read]

It's Official, Struts is History! (352): [read]

What Steve isn't telling us (348): So the rumors were true, Apple is really switching to Intel. There are a lot of interesting things in Steve Jobs' keynote, as usual, but the most interesting part is,... [read]

RE: Why I Ditched Hibernate (346): I saw this post and couldn't help but respond. The post's author, Bruce, is ditching Hibernate and Spring b/c he wants to use a connection pool (configured in Tomcat) instead.… [read]

How To Use Java With Blogger: A Tutorial

Tue, Apr 18, 2006 Blogger has a REST API. I use it to retrieve particular posts or to post transformed data. There is no Java API that I know of, but you will see here it is not very difficult to interface with Blogger API in Java using plain old XML.

Using libraries commons-httpclient and DOM4J it would be quite easy to implement your own Java Blogger API as the following code will suggest.

Authenticate
All requests need to be authenticated and are done in HTTPS. I use common-httpclient to perform requests. Here is how to setup the client:

private  HttpClient initHttpClient()
{
  HttpClient client  = new HttpClient();
  List authPrefs =  new ArrayList(2);
  authPrefs.add(AuthPolicy.DIGEST );
  authPrefs.add(AuthPolicy.BASIC);
  client.getParams().setParameter (AuthPolicy.AUTH_SCHEME_PRIORITY, authPrefs);
  client.getParams().setAuthenticationPreemptive(true);
  Credentials defaultcreds = new UsernamePasswordCredentials(user, password); 
  client.getState().setCredentials(new AuthScope( "www.blogger.com", 443, AuthScope.ANY_REALM), defaultcreds);
  return client;          
}

Get Your Posts
To retrieve the posts, you just have to query the right url, and parse the XML response. I prefer to use DOM4J, because of its handy asXML() method to print a node as XML. For simplicity I use a Map to store an XML entry.

public Collection getPosts() throws  HttpException, IOException, ParserConfigurationException, SAXException, DocumentException 
{
  GetMethod get = new  GetMethod("https://www.blogger.com/atom" +"/"+blogId);
  int  statusCode = client.executeMethod(get);
  if   (statusCode !=  HttpStatus.SC_OK)
  {
    throw new RuntimeException(" Could not make HTTP  request properly: " +get.getStatusLine());
  }
  InputStream response  = get.getResponseBodyAsStream();
  SAXReader reader = new  SAXReader();
  Document doc = reader.read(response);
  Collection posts = new ArrayList();
  List entries  = doc.getRootElement().elements("entry");
  if (LOG.isDebugEnabled()) 
  {
    LOG.debug("found  "+entries.size()+"  entries");
  }
  for (int  i = 0; i <entries.size();i ++)
  {
    Element entry  = (Element) entries.get(i); 
    Map m = new  HashMap();
    for (Iterator it = entry.elementIterator();it. hasNext();)
    {
      Element detail =  (Element) it.next();
      String name = detail.getName();
      if (name.equals("link"))
      {
        m.put("link ",detail.attribute("href").getValue());
      }
      else if (name. equals("content"))
      {
        m.put("content",detail.asXML());
      }
      else
      { 
        m.put(name,detail.getTextTrim());
      } 
    }
    posts.add(m);                   
    if  (LOG.isDebugEnabled())
    {
      LOG.debug( "found="+m.get(" title")+", url= "+m.get("link")); 
    }
  }
  return  posts;
}

Create XML for a new Post
Nothing particular here, just XML production.

private String createXmlForCreatePost(String postTitle, String postContent) throws   IOException, DocumentException
{
  SAXReader xmlReader =  new SAXReader();
  xmlReader.setValidation(false );
  Document doc = DocumentHelper.createDocument();
  QName rootName  = DocumentHelper.createQName("entry", new  Namespace("", "http://purl.org/atom/ns# "));
  Element root = doc.addElement(rootName);
  Element title = root.addElement("title"); 
  title.addAttribute("mode"," escaped");
  title.addAttribute("type ","text/plain");
  title.setText (postTitle);
  Element generator = root.addElement("generator ");
  generator.addAttribute("url" , "http://31416.org");
  generator. setText("31416 Java Generator ");
  Element content = root.addElement("content ");
  content.addAttribute("type" , "application/xhtml+xml");
  //Element  div = content.addElement(DocumentHelper.createQName("div",new  Namespace("","http://www.w3.org/1999/xhtml")));
  //div.add(...); //YOUR XHTML HERE
  StringWriter result = new  StringWriter();
  XMLWriter writer = new XMLWriter (result);
  writer.write(doc);
  writer.close();
  return result.toString();
}

That's it