Bi-Directional live Scrolling with Lazy Loading for PrimeFaces Datatable using Javascript

Primefaces Datatables are very versatile, but when it comes to certain features, the PF Team is very stubborn as not to release a feature unless it is needed by many. One such problem that I had was when I worked on the bidirectional scroll  feature for LiveScroll or the On-Demand Data. They already had the forward scroll implemented, but did not release a version for the backward scroll. So, I ended up making my own implementation. Here goes.

For simplicity, I built my table with only two columns. My model is as follows -

Car.java

public class Car {

    private String name;
    private String other;

    public Car(String name, String other) {
        this.name = name;
        this.other = other;
    }

    //Getters and Setters
}

It contains only two fields, name and other,  which are also the columns of my table. My lazy datamodel is as follows.

DataModel.java

public class DataModel extends LazyDataModel<Car> {

    List<Car> listVals;
    private List<Car> datasource;
    private int count = 200;
    private int backCount = 200;
    private String rowString = "";
    private static final long serialVersionUID = 1L;

    public DataModel() {
        datasource = new ArrayList<>();
        this.setRowCount(1000);
    }

    //Getter and Setter for rowString and listVals

    @Override
    public Car getRowData(String rowKey) {
        try {
            for (Car listVal : listVals) {
                if (listVal.getName().equals(rowKey)) {
                    return listVal;
                }
            }

        } catch (ArrayIndexOutOfBoundsException ex) {
            ex.printStackTrace();
        }

        return null;
    }

    @Override
    public Object getRowKey(Car car) {
        return car.getName();
    }

    @Override
    public List<Car> load(int first, 
                          int pageSize, 
                          String sortField, 
                          SortOrder sortOrder, 
                          Map<String, Object> filters) {
        listVals = new ArrayList<>();
        System.out.println("loading");

        int end = (count + 50);
        for (int i = count; i < end && count <= 5000; i++, count++) {
            listVals.add(new Car("lamborghini" + count, "other" + count));
        }

        datasource.addAll(listVals);

        return listVals;
    }

    public void loadPreCar() {
        listVals = new ArrayList<>();
        System.out.println("loading pre");

        int end = backCount;
         rowString="";
         
        for (int i = backCount - 50; i < end && backCount >= 0; i++, backCount--) {
            listVals.add(new Car("lamborghini" + i, "other" + i));
        }

        datasource.addAll(0, listVals);
       
        for (int i = 0; i < listVals.size(); i++) {
            Car car = listVals.get(i);
            rowString = rowString + "<tr class=\"ui-widget-content ui-datatable-even\" role=\"row\" data-ri=\"" + i + "\">"
                    + "<td role=\"gridcell\">" + car.getName() + "</td>"
                    + "<td role=\"gridcell\">" + car.getOther() + "</td>"
                    + "</tr>";
            
            i++;
            car = listVals.get(i);
            rowString = rowString + "<tr class=\"ui-widget-content ui-datatable-odd\" role=\"row\" data-ri=\"" + i + "\">"
                    + "<td role=\"gridcell\">" + car.getName() + "</td>"
                    + "<td role=\"gridcell\">" + car.getOther() + "</td>"
                    + "</tr>";
        }

    }
}

To perform lazy loading, it is important that the model class must extend the LazyDataModel<T> class. The load() method of the class is overridden so that we can compose the list on every callback and return it to the view. Here, it returns the next list when the bottom of the viewport/frame is reached. The loadPreCar()  method is where the customization is occurring. In this method, we generate the previous set of list occurring before the first record. This list is to be displayed when the backward live scrolling is performed.

The following is my bean.

DataBean.java

import java.io.Serializable;
import javax.annotation.PostConstruct;
import javax.enterprise.context.SessionScoped;
import javax.inject.Named;
import org.primefaces.model.LazyDataModel;

@Named
@SessionScoped
public class DataBean implements Serializable {

    private static final long serialVersionUID = 1L;
    private String value;

    LazyDataModel<Car>  data = null;

    public DataBean() {
    }

    public LazyDataModel<Car>  getData() {
        return data;
    }

    public void setData(LazyDataModel<Car>  data) {
        this.data = data;
    }

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }

    @PostConstruct
    public void init() {
        data = new DataModel();
    }
    
}

This class merely holds the datamodel and provides the getter and the setter for it.

Next comes my form.

testDatatable.xhtml

<h:form id="mainForm">
                <pf:growl id="growl"/>  

                <pf:panel>
                    <pf:dataTable 
                        var="car"
                        scrollable="true"
                        liveScroll="true"
                        scrollHeight="300"
                        scrollRows="50"
                        value="#{dataBean.data}" 
                        id="carTable" 
                        lazy="true" >
                        <pf:column headerText="Name">
                            <h:outputText value="#{car.name}" />
                        </pf:column>

                        <pf:column headerText="Other">
                            <h:outputText value="#{car.other}" />
                        </pf:column>
                    </pf:dataTable>
                </pf:panel>

                <h:inputHidden value="#{dataBean.data.rowString}"  
                               id="rowString"/>

                <pf:remoteCommand name="myRemote" 
                                  actionListener="#{dataBean.data.loadPreCar()}" 
                                  oncomplete="addRows()" update="rowString" />


</h:form>

If you notice, it contains three components - a datatable, a hidden variable and a remote command. The Javascript portion of the code is as follows -

<script type="text/javascript">
/* <![CDATA[ */

var lastScrollTop = 0;
var delay = (function() { //Adding delay
    var timer = 0;
    return function(callback, ms) {
        clearTimeout(timer);
        timer = setTimeout(callback, ms);
    };
})();

$(document).ready(function() {

    $('#mainForm\\:carTable .ui-datatable-scrollable-body').on('scroll', null, function() {
        var scrollLocation = $('#mainForm\\:carTable .ui-datatable-scrollable-body').prop('scrollTop');
        if (scrollLocation < 10) {
            var scrollB = $('#mainForm\\:carTable .ui-datatable-scrollable-body')

            if (scrollB.scrollTop() < lastScrollTop) {
                delay(function() {
                    myRemote();
                }, 300);
            }
            lastScrollTop = scrollB.scrollTop();

        }
    });
});



function addRows() {

    var rows = $('#mainForm\\:carTable .ui-datatable-scrollable-body table tr');

    for (i = 0; i < rows.length; i++) {
        var attrVal = parseInt(rows[i].getAttribute('data-ri')) + 50;
        rows[i].setAttribute('data-ri', attrVal);
    }



    var firstRow = $('#mainForm\\:carTable .ui-datatable-scrollable-body table tr:first');
    var rowHeight = firstRow.height();
    firstRow.before(document.getElementById('mainForm:rowString').value);

    var scrollB = $('#mainForm\\:carTable .ui-datatable-scrollable-body')
    scrollB.scrollTop('' + rowHeight * 50);
    lastScrollTop = rowHeight * 50;
}

/* ]]> */
</script>

It isn't as complex as it seems. Here, the scroll event simply fires the remote command method when the scroll bar is towards the top (for scroll position less than 10). The delay prevents the event from firing for every point in the scrollbar. The lastScrollTop variable keeps track of the direction of the scroll. The event fires only when the scrolling is happening in the upward direction.

The addRows() method adds the returned string from the backend to the beginning of the table. It also increments the data-ri, which is the row index attribute each row, by the number of newly added rows.


The flow -

When the page loads for the first time, the PostConstruct  in the DataBean.java creates an instance of the DataModel.java. As the table is loaded, it fires the load() method of the DataModel and creates the table with the first set of rows. Scrolling down functionality will behave in the expected way as for Primefaces. The Scrolling up functionality is where the magic occurs.

When the upper region of the datatable viewport/frame is reached, the js scroll event is fired which in turn triggers the remoteCommand to fire the method loadPreCar() in the class DataModel.java. This fetches the records and formats them into <tr> tags, thus forming a big string of records. This string is stored in the hidden variable rowString in the dataModel. The onComplete attribute in the remoteCommand tag fires the JS method addRows() after the execution completion of the backing bean. In this method, the generated string is retrieved from the hidden variable and is simply appended to the beginning of the table. A small adjustment to the row indices is done to all the existing rows.

This functionality works very similarly to the live downward scroll feature of primefaces. The add row method is of linear complexity to the size of the table. So, with a very large tables, performance may be affected due to the adjustment in the attributes of the existing  rows. Anyway, Hope this helps!

Parsing XMLs with SAX Parser

In my previous post (Refer: XML Reading and Writing using Servlets in Java), I used DOM Parser to parse the XML document. But, there is a very big disadvantage of the DOM Parser. It tries to load the whole XML file into the memory. With small XMLs it does not make much of a difference, but with big files, it can affect performance greatly. SAX on the other hand, parses an XML document in a stream fashion. So, it never loads the whole file into the memory and consequently its performance is much better.

Lets consider the following xml file -

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<first>
    <second atName="one">
        <number id="one">1</number>
        <number id="two">2</number>
    </second>
    
    <second atName="two"> 
        <number id="one">1</number>
        <number id="two">2</number>
    </second>
</first>

Now, the SAX parser mainly uses events to handle the data.  Which means the parser fires certain events as it parses through the documents (Source: saxproject.org). The most common events are as follows -

• Start Document - When the document begin is encountered
• Start Element - When the open tag for an element is encountered
• Charecters - This event reads the charecters between two tags
• End Element - When the end tag for an element is encountered
• End Document - When the document end is encountered

There are a few more events but these are the most commonly used ones. To use the functionality of these events, we extend the DefaultHandler class. 

public class SaxParser extends DefaultHandler 

This class helps us override the event methods. In my method GetXMLFile(), I have initiated the parsing.

private void GetXMLFile() throws SAXException, ParserConfigurationException, IOException {
        SAXParserFactory spf = SAXParserFactory.newInstance();
        SAXParser sp = spf.newSAXParser();
        sp.parse("src\\testXML.xml", this);
}

The first step would be to get an instance of the SAXParser from the SAXParserFactory. Then we simply feed the xml file to the parser which parses the file. It launches the following events which I handle in the following way.

The Start Document event -

@Override
public void startDocument() throws SAXException {
        System.out.println("Starting Parsing...");
}

The Start Element event -

@Override
public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
        StringBuilder sb = new StringBuilder();
        
        if (qName.equals("first")) {
            sb.append(qName).append("\n");
        } else if (qName.equals("second")) {
            sb.append("\t").append(qName).append(" ").append(attributes.getValue("atName")).append("\n");
        } else if (qName.equals("number")) {
            sb.append("\t\t").append(qName).append(" ").append(attributes.getValue("id")).append(" ");
        }
        
        System.out.print(sb.toString());
}

The charecters event -

@Override
public void characters(char ch[], int start, int length) throws SAXException {
    System.out.print(new String(ch, start, length).trim());
}

And the end element event -

@Override
public void endElement(String uri, String localName, String qName) throws SAXException {
        if (qName.equals("first")) {
            System.out.println(qName);
        } else if (qName.equals("number")){
            System.out.println("");
        }
}

The end document event -

@Override
public void endDocument() throws SAXException {
    System.out.println("Parsing Complete...");
}

The events fire according the the content of the xml and finally generates the following output -

Starting Parsing...
first
 second one
  number one 1
  number two 2
 second two
  number one 1
  number two 2
first
Parsing Complete...

Sax parsers are actually very powerful parsers and can be used very efficiently to go through a xml file.

I have attached the complete code just for reference.

package domsaxparser;

import java.io.IOException;

import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;

import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;

public class SaxParser extends DefaultHandler {

    public static void main(String[] args) {
        try {
            SaxParser xmlEditor = new SaxParser();
            xmlEditor.GetXMLFile();
        } catch (SAXException se) {
            se.printStackTrace();
        } catch (ParserConfigurationException pce) {
            pce.printStackTrace();
        } catch (IOException ie) {
            ie.printStackTrace();
        }
    }

    private void GetXMLFile() throws SAXException, ParserConfigurationException, IOException {
        SAXParserFactory spf = SAXParserFactory.newInstance();
        SAXParser sp = spf.newSAXParser();
        sp.parse("src\\testXML.xml", this);
    }

    @Override
    public void startDocument() throws SAXException {
        System.out.println("Starting Parsing...");
    }

    @Override
    public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
        StringBuilder sb = new StringBuilder();

        if (qName.equals("first")) {
            sb.append(qName).append("\n");
        } else if (qName.equals("second")) {
            sb.append("\t").append(qName).append(" ").append(attributes.getValue("atName")).append("\n");
        } else if (qName.equals("number")) {
            sb.append("\t\t").append(qName).append(" ").append(attributes.getValue("id")).append(" ");
        }

        System.out.print(sb.toString());
    }

    @Override
    public void characters(char ch[], int start, int length) throws SAXException {
        System.out.print(new String(ch, start, length).trim());
    }

    @Override
    public void endElement(String uri, String localName, String qName) throws SAXException {
        if (qName.equals("first")) {
            System.out.println(qName);
        } else if (qName.equals("number")) {
            System.out.println("");
        }
    }

    @Override
    public void endDocument() throws SAXException {
        System.out.println("Parsing Complete...");
    }

}

Have fun!!

Saving Memory with Primitive ArrayLists, HashMaps and BitSets in Java

Primitive type Arraylists

This summer, when I was trying to develop a small scale database, I faced a huge memory problem. I tried to keep the columns in an ArrayList, which was the obvious choice as they can increase dynamically in size. But, on the downside, ArrayList can only work with objects. So, int would have to be used as Integer, double as Double and so on.

The problem with this approach was, as soon as the primitive types are wrapped by objects, their sizes triples. For instance, here is an illustration to show the difference between double and Double. (Source: University of Virginia)

Now, this representation ofcourse varies from JVM to JVM, but this is the idea. The overheads are mostly caused by pointers and object metadata. This restricted me from storing a lot of these values. (oh! It was a Main Memory database, so forget about writing the values to disk). Also, with more space consumption, my GC overhead increased considerably. Yeah! I know!! I need a powerful machine. :(

Anyway, this got me looking up a bit till I stumbled upon the Apache library who created ArrayList versions for primitive files. Here is a link to it -

Primitive Lists

Now, this class contains primitive lists for almost all primitives like ArrayIntList, ArrayDoubleList, ArrayCharList etc. It even contains ArrayBooleanList. It has the same methods and functions exactly like the arraylist.

Bitset

The success with integers made me wonder, can I do better with the boolean? You see, boolean primitives take up size between 1 byte to 4 byte depending upon the JVM and the way it is implemented, but it actually needs only 1 bit to store the information. So, to store a list of booleans, we can actually use a BitSet. This grossly reduces the memory footprint thereby solving my memory problem upto a great deal.

I have a small example of how bit sets can be used as boolean lists.

import java.util.BitSet;

public class testBitSet {
    public static void main(String[] args) {
        BitSet bs = new BitSet();
        
        for(int i=0;i<20;i+=2){         //Sets alternate bits to true
            bs.set(i);
        }
        
        //Randomly Printing bits
        System.out.println("Element 11: " + bs.get(11));    //false
        System.out.println("Element 12: " + bs.get(12));    //true
        
        //Changing value of element at position 12 to false
        bs.set(12, false);
        System.out.println("Element 12: " + bs.get(12));    //false
        
        //printing all set element. Prints [0 2 4 6 8 10 14 16 18 ]
        int cursor=bs.nextSetBit(0);
        System.out.print("All set elements : [");
        while(cursor!=-1){
            System.out.print(cursor+" ");
            cursor=bs.nextSetBit(++cursor);
        }
        System.out.println("]");
    }
}

Primitive type Hashmaps

Another interesting set of collections are the Maps or the HashMaps which, again, take objects as references. If only they could take primitive types :(. Well luckily, they can be modified to do this too :). The classes from the Trove package does exactly this. They contain different classes like TIntIntMap,TDoubleIntMap or TObjectIntMap (Which can take strings/objects as keys). Needless to say they work in the same way as regular util maps. Here is a link to it -

Gnu Trove Package

Hope this helps people out there trying to save memory.

Difference between .equals and == in java

In Java, == checks for reference equality while .equals() checks for content equality, which is true in every case. But, java refers to the memory in some inconsistent way which brings up all the confusion.

Check the following code -

class you{
    public static void main(String[] args){
        
        Integer a = 1;
        Integer b = 1;   // a & b are pooled from the same memory location
        Integer i = 128;
        Integer j = 128;  /* i and j are not pooled. Hence stored in different memory location */
        
        System.out.println("a == b: "+(a == b));  //true
        System.out.println("a.equals(b): "+(a.equals(b)));  //true
        System.out.println("i == j: "+(i == j));  //false
        System.out.println("i.equals(j): "+(i.equals(j)));    //true
        String x="abc";
        /* Since strings are immutable, x and y are stored in the same location in the heap memory for same value*/
        String y="abc";
        /*Creating a new reference. z has a  different memory location but same value*/
        String z=new String("abc");
        System.out.println("x == y: "+(x == y));  //true
        System.out.println("x.equals(y): "+(x.equals(y)));   //true
        System.out.println("x == z: "+(x == z));  //false
        System.out.println("x.equals(z): "+(x.equals(z)));   //true
    }
}

       For Integer objects, when the value is below 8-bits(-128 to 127), java stores it ,for the same values, in the same location in the memory and pools it from there. Hence all objects point to the same memory location for these values. But when the value is out of this range, dedicated memory locations are created and hence at those times == returns false.

       For Strings, since they are immutable, java stores them in the same memory location (irrelevant of size) when declared directly. However, if u create a string type object and then assign the same value, the memory location is diff.

So its always safer to use .equals() when u want to do content checking for strings.